System Operations: the Java side (part 2)
Recap
● OutSystems Platform Architecture○ Platform Services○ 1 Click Publish○ Standalone vs Farm Environments
● OutSystems Platform Licensing○ Activation Code○ Serial Number○ Intellectual Property Protection (IPP)
● Installation & Configuration○ Versioning○ Checklist○ Configuration Tool
System Operations: the Java side (part 2)
Farm installation
● What is different in the farm installation?○ Multiple machines work together to serve or deploy the applications - multiple
Front-ends;○ Only one Deployment Controller – Deployment Controller Service is only
running on one machine;○ Start by configuring the Deployment Controller, then export the configuration file
to be imported in the Front-Ends
● What about adding a front-end to an existing farm?○ Applications are automatically deployed the moment Deployment Service is
started;○ Same installation procedure – same base server.hsconf
System Operations: the Java side (part 2)
Farm installation
● What is different in the farm installation?
Controller address
Front End address
System Operations: the Java side (part 2)
Farm installation
● Farm installation overview○ Similar procedure to standalone, having in mind the profile of each machine○ License, Service Center, System Components etc. need only be published to the
first Front-End machine
● Differences from Standalone installation○ In Configuration Tool:
■ Configure in Deployment Controller machine first■ Deployment Controller IP - not localhost anymore!■ Copy resulting server.hsconf file■ On every other machine, paste server.hsconf before running Configuration
Tool
System Operations: the Java side (part 2)
Configuration
● Load Balancers○ Up to Layer 7 Load Balancers○ Most common balancing algorithms (round-robin, least connections, fastest
HTTP response, etc …)○ Both sticky or non-sticky sessions persistent due to the shared session data
model○ HTTP to HTTPS tunneling
● Compressed content (static and dynamic)
● Content caching (expiration headers)○ From 9.1 on, JBoss/WildFly stack only○ Cache suffix for static contents
System Operations: the Java side (part 2)
Farm installation
● Network Requirements (connectivity)Publishing and Runtime
Source Destination Port Protocol Notes
Controller Oracle / MySQL 1521/3306 TCP Database Connection
Controller Front-End 12001 TCP OutSystems Deployment Service
Front-End Oracle / MySQL 1521/3306 TCP Database Connection
Front-End Controller 12000 TCP OutSystems Deployment Controller Service
Controller Front-End 2033 TCP RMI (needed for OutSystems Services)
Front-End Controller 2033 TCP RMI (needed for OutSystems Services)
Controller Front-End 5556 TCP Node Manager (WebLogic only)
Front-End Controller 7001 & 7002 TCP Admin Server (WebLogic only)
External Users DMZ Front-End 80 TCP Application's HTTP access
External Users DMZ Front-End 443 TCP Application's HTTPS access
Internal Users Internal Front-End 80 TCP Application's HTTP access
Internal Users Internal Front-End 443 TCP Application's HTTPS access
System Operations: the Java side (part 2)
Farm installation
● Network Requirements (connectivity)Monitoring
Source Destination Port Protocol Notes
Front-End Controller 2033 & 12000 TCP RMI/OutSystems Deployment Controller Service
Front-End Controller 2033 & 12003 TCP RMI/OutSystems Log Service
Front-End Front-End 80 TCP Application Server
Front-End Front-End 2033 & 12001 TCP RMI/OutSystems Deployment Service
Front-End Front-End 2033 & 12002 TCP RMI/OutSystems Scheduler Service
Front-End Front-End 2033 & 12003 TCP RMI/OutSystems Log Service
System Operations: the Java side (part 2)
Connectivity clarification
● Jboss/WildFly/WebLogic are normally configured to listen to ports 8080 and 8443○Instead of the default Web 80 and 443 ports○Reason: Security limitation - Linux restricts binding of ports under 1024 to
priviledged/root users
● IPTables (firewall) should be configured to forward Web traffic to the Application Server ports
○Port 80 to 8080 (HTTP)○Port 443 to 8443 (HTTPS)
System Operations: the Java side (part 2)
Upgrading
● OutSystems Platform Server upgrade (changing M.m):
1. Backups!2. Pre-requirements check up and installation;3. Install the Platform Server software;4. Upgrade the metadata;5. Install Service Center6. Install System Components7. Upgrade the applications
● Development tools are upgraded as well
System Operations: the Java side (part 2)
Upgrading
● Application upgrade○ In Development, do in-loco component upgrade
■ Validate documented breaking changes;■ Upgrade extension modules first;■ Upgrade all eSpaces;■ Assistance from developers!
○ In Production, use a previously upgraded solution!■ Faster and safer!■ Applications typically require changes after an upgrade;■ Operation can be done by operators■ Can be done with zero downtime!
System Operations: the Java side (part 2)
Patching
● A patch is a light update:○ The release and/or revision changes (M.m.R.r);○ Development tools don’t change;○ No breaking changes;○ No additional software requirements.
● Steps:○ Install Platform Server;○ Update metadata and install Service Center;○ Update System Components○ Republish all applications.
● Should be applied in all staging environments before applying in Production!
System Operations: the Java side (part 2)
Tuning
● Front-End Performance Tuning○ OutSystems Platform○ Application Server○ Applications
System Operations: the Java side (part 2)
Tuning – OutSystems Platform
● The only tuning option is enabling/disabling Debug Mode (in eSpaces)○ If environment Running Mode is Production, OutSystems Platform enforces
Debug Mode OFF in eSpaces;○ If environment Running Mode is Development, you can choose the Debug Mode
for eSpaces■ Should be ON in Development environments;■ Should be OFF in Pre-Production environments, to be as close to the
Production Environment as possible.
● Advanced tuning is possible at connection string level:○ “If it ain’t broken, don’t fix it!”
System Operations: the Java side (part 2)
Tuning – Application Server
● JBoss memory limits
● JBoss is highly configurable via .conf files○Though OutSystems recommendation is to stick to configuring the Xmx memory
only
System Operations: the Java side (part 2)
Backups
● What should I backup?○ The OutSystems Database
○ The Configuration Tool export file
○ Any filesystem, Application Server customizations
System Operations: the Java side (part 2)
Platform Internals
● 1-Click Publish● 2-Stage Deployment● LifeTime● Timers and asynchronous processes● Logging mechanism
System Operations: the Java side (part 2)
Platform Internals
● 1-Click Publishing Functional Process
Service Center Deployment Controller Deployment
1. Sends OML to Controller
1st Stage Compilation
2nd Stage Compilation
Pack Application
2. Compile eSpace
3. Broadcast Application Files
5. Deploys Application in App Server
4. Update DB
XML
Java ProjectOracle
JSP/JSFWAR
ZIP
ZIPOML
OML
JAR
System Operations: the Java side (part 2)
Platform Internals
● 1-Click Publish:○ One single file (OML) contains the complete application definition (compressed
XML)○ Deployment across several front-ends farm (no manual intervention)○ Unified deployment procedure for both application code and database scripts
(no manual intervention)○ External dependencies are deployed with the application (no separate
installation procedures)○ Deployment actions are logged for auditing purposes
System Operations: the Java side (part 2)
Platform Internals
● 1-Click Publishing Deployment○ Executed across all active Front-ends without manual intervention;
○ eSpaces are deployed to the running folder inside the OutSystems Platform Server directory – typically /opt/outsystems/platform/running;
○ Upon startup, each Deployment Service communicates with the Controller to ensure that it is up-to-date – and if not, begins deployment of eSpaces needing updates.
System Operations: the Java side (part 2)
Platform Internals
● 1-Click Publishing Deployment○ Parallel File Distribution
■ Only modified files will be broadcasted in parallel to Front-Ends
○ Front-End Shared Server Library■ Compiled assemblies (JAR) are stored in a shared repository in the Front-End
System Operations: the Java side (part 2)
Platform Internals
● 2-Stage Deployment
○ Allows to execute the long and heavy lifting operations in a preparation stage (1st stage - compilation and file deployment) without impacting application availability
○ Perform the application deployment in a narrowed time window (2nd stage – data model update and Application Server runtime directory updates) using hot deployment
System Operations: the Java side (part 2)
JSP/JSFWAR
Platform Internals
● 2-Stage Deploy - 1st Stage Functional Process
Service Center Deployment Controller Deployment
3. Broadcast Application FilesZIP
1. Sends OSP to Controller OSP
OSP
1st Stage Compilation
2nd Stage Compilation
Pack Application
2. Compile eSpace
XML
Java ProjectOracle
ZIP
JAR
System Operations: the Java side (part 2)
Platform Internals
● 2-Stage Deploy – 2nd Stage Functional Process
Service Center
3. Broadcast Application FilesZIP
1. Sends OSP to Controller OSP
OSP
1st Stage Compilation
2nd Stage Compilation
Pack Application
2. Compile eSpace
XML
Java ProjectOracle
ZIP
JAR
Deployment
5. Deploys Application in App Server
4. Update DB
Deployment Controller
JSP/JSFWAR
System Operations: the Java side (part 2)
Platform Internals
● LifeTime Architecture
○ It’s an OutSystems system application■ Web Application
○ Runs only on one environment
○ Deploys applications across environments
○ Requires that all environments are on the same version (M.m)
System Operations: the Java side (part 2)
Platform Internals
● LifeTime on Production environment
Development Test Production
LifeTime User
System Operations: the Java side (part 2)
Platform Internals
● LifeTime on it’s own environment
Development Test Production
LifeTime User
System Operations: the Java side (part 2)
Platform Internals
● LifeTime Synchronization process
○ Keeps LifeTime application consistency and version information up to date
○ Occurs after every publication, and periodically
○ Uses Business Process Technology (BPT)
System Operations: the Java side (part 2)
Platform Internals
● Scheduler service is responsible for○ Running Timers○ Executing BPT events and process activities○ Sending Emails
App n
.
...
App 2App 1
queue
Main database
Front-endScheduler Service
Timer thread
Process thread
Timer thread
Timer thread
Process thread
…
Process thread
Emailthread
Emailthread
System Operations: the Java side (part 2)
Platform Internals
● Scheduler and timers○ The Scheduler checks timers to run on the database○ Evaluate the NEXT_RUN value against current date○ Locks the Timers data model and update the timer status to RUNNING○ This lock allows multiple OutSystems Schedulers to concurrently access the same
table, without running the same timer at the same time in different Front-Ends○ Executes timer with a local Web Service Request○ Upon completion, if there’s a schedule, redefines the NEXT RUN value
System Operations: the Java side (part 2)
Platform Internals
● Scheduler and BPT processes○ The Scheduler checks events and process activities to execute on the database○ Evaluate the NEXT_RUN value against current date○ Locks the event or activity data model and update the event status ○ This lock allows multiple OutSystems Schedulers to concurrently access the same
table, without running the same event or activity at the same time in different Front-Ends○ Executes process activities associated with the event, using a local Web Service
Request
System Operations: the Java side (part 2)
Platform Internals
● Scheduler and Emails○ The Scheduler checks the email queue on the database○ Evaluate the NEXT_RUN value against current date○ Locks the email data model and update the email status as RUNNING○ This lock allows multiple OutSystems Schedulers to concurrently access the same
table, without running the same email at the same time in different Front-Ends○ Sends the emails directly from the Scheduler Service process and updates it’s
status○ Retry mechanism
System Operations: the Java side (part 2)
Platform Internals
● Logging○ OutSystems Platform database logs are designed to scale with thousands or
millions of logs;○ Each log entry is stored in a table from a set which is used in sequence
■ Log tables go from oslog_<TYPE>_0 through oslog_<TYPE>_9;■ For each type, a view is mapped to the current table in the rotation
(oslog_<TYPE>) and one to the week before (oslog_<TYPE>_Previous)○ Log writing is executed directly to the respective table, to improve performance;○ Log reading in Service Center uses the two views;
■ Current Week■ Previous Week
○ One sequence of log tables for each log type.
System Operations: the Java side (part 2)
Platform Internals
● Logging○ Log rotation occurs weekly – every Friday at 11h45 PM;○ The log table used in each week uses a fixed algorithm:
■ Number of weeks since Jan 1st 2000 MOD 10;○ Every week the next table in each sequence is used, rolling over when necessary;○ The views for each log type are updated upon rotation;○ Logs from previous weeks are still in the database, though not accessible
through Service Center ■ Can be obtained by querying the tables directly;
○ Old log tables are cleaned after their retention period passes – preparing them for later use.
System Operations: the Java side (part 2)
Platform Internals
● Logging
Log Database
Table 0Table 1Table 2
Table 8Table 9
.
.
.
Views
Log Service
Service Center OutSystems Applications
Log Read accessLog Write AccessCurrent DB View AssociationPrevious DB View Association
Message Queue
Services
System Operations: the Java side (part 2)
Troubleshooting
● Basic troubleshooting techniques
● Advanced troubleshooting techniques
System Operations: the Java side (part 2)
Troubleshooting
● Basic troubleshooting: Get the error
○ Check Environment Health
○ Analyze Service Center logs
○ Analyze OutSystems Services Logs
○ Analyze Application Server Logs
System Operations: the Java side (part 2)
Troubleshooting
● Basic troubleshooting: Environment Health
System Operations: the Java side (part 2)
Troubleshooting
● Basic troubleshooting: Service Center log
System Operations: the Java side (part 2)
Troubleshooting
● OutSystems Log Files○ Each Platform Service has its own log (found in /opt/outsystems/platform/logs)
■ Includes OSTraces (if enabled)■ Logs rotate everyday (One file per day)
○ To configure OSTrace logging level■ Edit config file /etc/outsystems/os.<service_name>.service.properties■ Change the log4j.logger.outsystems setting to the intended level (FATAL,
ERROR, WARN, INFO or DEBUG)
System Operations: the Java side (part 2)
Troubleshooting
● Application Server Logs (JBoss/WildFly)○ All found in $JBOSS_HOME/standalone/log/
■ e.g. /opt/wildfly-8.2.1.Final/standalone/log/
○General Log: <logpath>/server.log■Contains log4j logs from both the Application Server and the OutSystems
Platform
○Access Log: <logpath>/access_*.log
○General logs rotate with date suffixes (e.g. server.log.2015-12-25)
System Operations: the Java side (part 2)
Troubleshooting
● Application Server Config (JBoss/WildFly)○ All found in $JBOSS_HOME/standalone/bin/
■ e.g. /opt/wildfly-8.2.1.Final/standalone/bin/
○App Server instance: <configpath>/standalone-outsystems.conf■Contains log4j logs from both the Application Server and the OutSystems Platform
○Message Queues instance: <configpath>/standalone-outsystems-mq.conf■Contains log4j logs from both the Application Server and the OutSystems Platform
System Operations: the Java side (part 2)
Troubleshooting
● Application Server Logs (WebLogic)○ All found in
$WL_HOME/user_projects/domains/outsystems_domain/servers/<managedserver>/logs/■e.g.
/opt/Oracle/Middleware/wlserver/user_projects/domains/outsystems_domain/servers/ip-10-142-241-8/logs/
○ General Log: <logpath>/<managedserver>.log■ Contains log4j logs from both the Application Server and the OutSystems Platform
○ Access Log: <logpath>/access.log
○ Both these logs rotate with numeric suffixes (e.g. access.log00123)
System Operations: the Java side (part 2)
Troubleshooting
● Basic Troubleshooting: Working the Error○ OutSystems Troubleshooting Guide1. Search “Troubleshooting the OutSystems Platform Server” according to error
description2. Validate possible “Cause” as described;3. Execute “Recovery Action”;
○ Search Internet Information Knowledge Bases■ Search for the error message and description■ Confirm that the scenario matches the error description■ Evaluate and apply recommended actions
NOTE: Always consider certified internet knowledge bases!
System Operations: the Java side (part 2)
Troubleshooting
● Advanced Troubleshooting: Get the error○ System Performance Monitor○ System network tools
■ ifconfig■ ping■ netstat■ telnet
○ Network protocols sniffers (YATT, Wireshark, etc)
System Operations: the Java side (part 2)
Troubleshooting
● Advanced Troubleshooting: Working the error
○ Search Internet Information Knowledge Bases■ Search for the error message and description■ Confirm that the scenario matches the error description■ Evaluate and apply recommended actions
NOTE: Always consider certified internet knowledge bases!
System Operations: the Java side (part 2)
Troubleshooting
● Proactive Monitoring and prevention○ Proactive monitoring can help prevent problems by exposing them before they
reach a critical state○ A simple monitoring approach should consist of:
■ Check Platform Monitoring screen ■ Check Platform Server logs■ Check LifeTime Performance Analytics■ Check Infrastructure Monitor
○ A more in-depth monitoring approach should also include:■ Check the reports available in Service Center■ Check System logs messages for each node
Next webinars
April 7 - "Detect performance bottlenecks (Performance CSI)" - with Paulo Garrudo
April 21 - "Building a Live Style Guide" - with Ruben Gonçalves
Top Related