09_AppDevAdminLifecycle

® IBM Software Group © IBM Corporation DataPower App Development and Admin Lifecycle IBM Software Group | WebSphere Software / DataPower © IBM Corporation Overview � Centralized Development �Test/Version Control/Build Approach �Version Control Issues � Centralized Device Management �Sources & Pull/Push �Zip vs XML vs CFG �Include vs Import �Approaches �Other Options �Other Notes �Imminent/Future Relief IBM Software Group | WebSphere Software / DataPower © IBM Corporation Centralized Device Management – Zip vs XML vs CFG Formats �Export/Import Zip Bundles (*.zip) � Only practical at the domain level and below, you can't conveniently use them to store inter-related sets of objects in different domains – Preferred/recommended approach from DataPower engineering – Contains extra files with info on firmware-specific things, device config info, task templates, etc. – Extra intelligence for picking up things when going across versions of firmware �Export/Import XML Files (*.xcfg) � Representation of object(s) in XML � As you would expect, being XML, this is quite verbose/large � Not as well tested/supported as Zip format, an “alternate” approach �Config files (*.cfg) � Look under File Management, “config:” folder – For default domain, autoconfig.cfg. For other domains, domain_name.cfg – These are built/changed for you when you save your configuration � You can create these on your own, they are just sets of CLI commands – DP support frowns on this, hand-edited configs may not be supported IBM Software Group | WebSphere Software / DataPower © IBM Corporation Centralized Device Management – Sources & Pull/Push �Objects->Configuration->Import Configuration � Was named ‘Import Package’ prior to 3.6.1 � This gets an XML or ZIP file from the source URL and imports the contents � Gets the entire package on *each* device reload/restart, set options to overwrite files or config, no warnings about conflicts � Has the ability to also include resource files such as XSLT stylesheets, FFD, etc �Objects->Configuration->Include Configuration � This specifies CFG files to be pulled from the source URL on *each* reload � Can only include configuration information, and does not have the ability to also include resource files (XSLT, FFD, etc) �Administration->Configuration->Import Configuration – Import XML or ZIP configuration data from a remote file – One-time shot, GUI/wizard driven, prompts for overwrite – Can select specific objects in config file to import – Modifies local configuration file � Be careful about introducing port conflicts when bringing these in! IBM Software Group | WebSphere Software / DataPower © IBM Corporation � Remote Configuration for Application Domain �Preferred approach as far as granularity Centralized Device Management – Sources & Pull/Push IBM Software Group | WebSphere Software / DataPower © IBM Corporation Centralized Development - Test/Version Control/Build Approach � Developers use their own domains, one per developer, unit test �Check any individual artifacts/objects built into version control � Front End Handlers, AAA Policies, Match Rules, etc - components � These are somewhat equivalent to Java source files in WAS/J2EE build scenarios � Now do integration testing at the service level (i.e. WS-Proxy, XML F/W) �Combine things needed and related, i.e Front End Handlers, AAA policy � Pull these (from prior development step) from source control �Check results in now as higher level (service) objects to version control � These will include the lower-level objects used by the service (AAA policy, etc) � So, these are somewhat equivalent to war, jar, ejb modules in WAS/J2EE builds � Combine services into top level application domains and test �Check into version control, these are somewhat similar to EAR files in WAS/J2EE builds � Ideally version control and logical service granularity would be organized on a per application domain level � All of the above is for the DataPower configuration files. The separate files that you use for things like XSL stylesheets, FFDs, can remain in version control and not be included in these config bundles if you wish. �You can reference/pull these at run-time from DataPower (detail in subsequent slides) IBM Software Group | WebSphere Software / DataPower © IBM Corporation Centralized Development - Version Control Issues � Put version numbers in the file name or inside the files? � A classic debate, not covered here, but embedded comments seem to be favored more by “right thinking” people � Labels are your friend � The easiest scenario is to use a source control system that supports "get by label" via http IBM Software Group | WebSphere Software / DataPower © IBM Corporation Centralized Device Management – Sources & Pull/Push � Where to get configuration from when building environments �Manual import - not acceptable for repeated process due to likely human error �XML Mgmt Interface (SOMA) – see later example using curl � Use a “roll your own” client – Web Services interface to DP, usually port 5550 � Used by Eclipse plugin � Interface implies much detail, may change with each version of firmware � Closely tied to underlying object model which complicates versioning � Not well documented - store:///xml-mgmt* files contain the WSDL and schemas for any requests that can be sent to SOMA � Hence, use for actions and status, don’t use SOMA for config gen (i.e. building XML Firewall, etc) �See following slides for more options � Where to pull resources (XML, XSL, FFD) from at run-time � DataPower local file system (i.e. local:xxx.xsl) is fastest, easiest � HTTP Server – Could be an http interface provided by source/version control product. DP uses urls to get to things, so just use http://xxxx. Leverages caching on http server and DataPower. � NFS Mount (shared drive) – DP can understand/see these but this is not recommended due to overhead of static TCP connections used for NFS � Can use the CLI ‘copy’ command to pull these at startup/reload and into cache � DataPower will cache these types of files for you, so they won’t need to be pulled each time � Configure caching policies on XML Manager objects IBM Software Group | WebSphere Software / DataPower © IBM Corporation Centralized Device Management – Sources & Pull/Push Eclipse/RAD Plug In (push to device): Central application running on Administrator workstation Can change/update/distribute Configuration and Firmware for multiple boxes Provides real-time status information for multiple boxes Uses XML Mgmt Interface (SOMA) IBM Software Group | WebSphere Software / DataPower © IBM Corporation Centralized Device Management – Sources & Pull/Push � Using XML Mgmt Interface (SOMA) (push to device) SOAP calls to XML Mgmt interface capable of configuring all aspects of device SOAP calls can return all status information Often requires custom development Subject to RBM Access Policies Can send from custom Web Service client, http via curl, or python script Used by Eclipse/RAD DataPower plugin Not meant for config generation Use for actions or status commands IBM Software Group | WebSphere Software / DataPower © IBM Corporation Centralized Device Management – Sources & Pull/Push � Remote Configuration Servers (pull from device) Device will “pull” config from remote configuration servers (Include Config, Import Pkg, Import Config) Can include all or part of the configuration of all or part of the device Use from Web GUI or CLI Some are dynamic, some are a ‘one time shot’ (pulled once and then becomes a local config) See next slide for details IBM Software Group | WebSphere Software / DataPower © IBM Corporation Centralized Device Management – Sources & Pull/Push � Other options to push config and get device status: �Standard operating system shell script (.sh or .bat) � Take in params on command line, build response file, send ssh command with pipe to response file input �Python scripts � Can build SOMA XML file, send via curl IBM Software Group | WebSphere Software / DataPower © IBM Corporation Centralized Device Management - Approaches � Example Scenario 1 – Exported XML Config (xcfg) using CLI �Develop services that need to share various objects – i.e. load balancer, AAA policy object, MQ Queue Manager, HTTP front side handler �Divide up the packages appropriately, ending up with several exported XML packages (xcfg) �Do Import Package on each package � Whoops! You may find that these are executed asynchronously! – You can immediately see this was unworkable because the packages depend on one another and therefore load order is critical � The work-around is to Import Package the first package – The final object in the first package is an ImportPackage of the next package, and so on, producing a linked list that imposes the order needed to ensure the dependencies are satisfied – A bit tedious and error prone in large shops – Inherently racy and cumbersome IBM Software Group | WebSphere Software / DataPower © IBM Corporation Centralized Device Management - Approaches � Example Scenario 2 – Dynamic Config (cfg) files � You have a CLI script that looks like this (simple example): matching "matchAll“ urlmatch "/*“ exit � Put on the web server or source control as “matchall.cfg” � Edit your domain_name.cfg file to say: exec "https://webserver/matchall.cfg" � The exec statement behaves as a #include and the matching rule will be created at that point in the initialization process � You don't need the SOMA interface for this technique, just directly edit the config file for the domain � Careful! Once you've edited the config file directly you can't use the Save Config button in that domain again because it will overwrite your changes � But this is for non-dev domains, where changes aren’t as frequent and are usually scripted/automated, and Web GUI is often not allowed � Web GUI can still be used for monitoring � As a “supported” alternative, use the Include Configuration option from the Web GUI to pull these in, and use higher level objects, i.e complete services rather than lower-level objects � The 3.6 firmware improves this a great deal by remembering whether an object was loaded from an external file and prompting you before you overwrite the domain config file IBM Software Group | WebSphere Software / DataPower © IBM Corporation Centralized Device Management - Approaches � Example Scenario 3 – Scripted Config (cfg) files � Build scripts, for example domain.cfg: �Create command macros/alias for common tasks such as copying files (should use https!): IBM Software Group | WebSphere Software / DataPower © IBM Corporation Centralized Device Management - Approaches � Example Scenario 4 – WAS/J2EE-type build/deploy � May not be supported, this is theoretical � You have cfg or xcfg files representing the lower level objects checked into version control � This is the same as scenario 2 so far � When all pieces are checked and you are ready to build the final config, use a build tool such as Ant to pull these lower level objects’ config files together to create a domain_name.cfg (or xcfg) and check into version control � This is now equivalent to your finalized ‘EAR’ file in a J2EE scenario. � From here, you can have DataPower point to this config remotely for the domain � Alternative might be to import domain config in autoconfig.cfg on device startup with ‘exec’ IBM Software Group | WebSphere Software / DataPower © IBM Corporation Centralized Device Management - Approaches � Example Scenario 5 – Scripted admin with Python �See the AddDomain.py, send-files-to-dp.sh/py example files � Takes parameters for domain, group, user, etc � Pops up entry dialog if params not provided � Writes out an XML SOMA file and submits via curl in a pop-up window � Creates notification to user that their domain is created IBM Software Group | WebSphere Software / DataPower © IBM Corporation Centralized Device Management - Approaches � Example Scenario 6 – Scripted admin with shell scripting �See the copy-files-to-dp.sh files �Creates CLI command response file �Opens SSH session, submits �Greps response message IBM Software Group | WebSphere Software / DataPower © IBM Corporation Centralized Device Management – Other Options � Set up the device to pull its entire config from a central server/source control at startup using statements in the autoconfig.cfg file �Startup deck for each device could contain static info such as networking data (i.e. IP, DSN) and fetch all application data from source control system using ‘exec’ command shown in previous slide. � Set up domains to pull their config from remote servers on restart using this same technique (see prev slide) � To confirm successful startup, set up a confirmation task after the initiation of the device configuration �Establish an echo service match rule to return a health-check document that would confirm the success of the configuration IBM Software Group | WebSphere Software / DataPower © IBM Corporation Centralized Device Management – Other Notes � In general, Unix people are much more comfortable with the CLI/cfg approach while Windows people seem to prefer the XML packages � It is a very good idea to log out frequently when developing configuration stuff. The Import/Export Configuration wizards use lots of ramdisk, as does Restart Domain. Use too much and your box will behave erratically. � DNS/IP Address substitution (dev, prod, etc) �Using DNS and host names � Device startup deck can designate DNS server, or individual DNS entries – Sometimes not a good option, often one DNS server has all entries for environments � Use Host Alias on DP device under Network settings to avoid putting IP addresses in DP �Can use Ant, XSLT or text/string file parsers to substitute in files before deploy �Automatic local-ip substitution not available through SOMA or Web GUI yet, but will come in a future release. This should solve 90% of all issues with local ip � It’s best to not get too carried away with very granular separation of objects from an application domain. The application domain is considered by DataPower to be the ‘independent’ entity and the level at which reuse should be done. Ideally version control and logical service granularity would be organized on a per domain level. IBM Software Group | WebSphere Software / DataPower © IBM Corporation Centralized Device Management – Other Notes � Certain things can’t be ‘exported’ � Private keys in the cert: directory � If generated/obtained off-box, you should have copies from the source � If generated on-box, if you didn’t click the radio button to export then - sorry! � User identities, ACLs � Best way to do this is to build scripts to add users/groups to a device � Can be done ad-hoc with CLI, or by using the XML/SOMA approach discussed earlier � Note the curl command to send this (embedded as a comment) IBM Software Group | WebSphere Software / DataPower © IBM Corporation Centralized Device Management – ITCAM SE for DataPower � ITCAM System Edition for DataPower (v3.6 and up) � New product from Tivoli/DataPower, free for customers IBM Software Group | WebSphere Software / DataPower © IBM Corporation Centralized Device Management – Imminent/Future Relief � WAS Admin Console Plugin for DataPower �A war file that you install to the admin console �Alpha works tech preview technology: http://www.alphaworks.ibm.com/tech/dpwas � Lists devices in the enterprise, click to get to the console for any of them �WAS 6.0 and up, DataPower 3.5.1.3 and up � DataPower Admin through WAS Admin Console �Polished, ‘productized’ version of the above �V.next WAS major release (v7?) � Solutions based on WAS XD? � DataPower v3.6 config diff’ing tool IBM Software Group | WebSphere Software / DataPower © IBM Corporation Configuration Comparison II Enhanced Configuration Comparison includes ability to do the following: • Determine Checkpoint Limit • Set Checkpoints per Application Domain • Roll Back Application Domain to Checkpoint • Compare Checkpoint Config to Other Targets New in 3.6.1 Slide by David Shute IBM Software Group | WebSphere Software / DataPower © IBM Corporation Deployment Policy A Deployment Policy provides mediation for imported configuration files from a remote appliance to a local appliance, or from exported configuration packages. The Deployment Policy includes a statement [match statement, domain, resource, access level, property and value]. The metadata churned from a Deployment Policy of the type: • Accepted can be added to a configuration that is being imported. • Filtered can be deleted from a configuration that is being imported. • Modified can be added, deleted or changed in a configuration that is being imported. New in 3.7.1 Slide by David Shute IBM Software Group | WebSphere Software / DataPower © IBM Corporation Deployment Policy (cont’d) Accepted and Filtered statements on a Deployment Policy: • Filename value of Robbie in the statement of a Deployment policy of the type Accepted. • Filename value of Riba in the statement of a Deployment policy of the type Filtered. New in 3.7.1 Slide by David Shute IBM Software Group | WebSphere Software / DataPower © IBM Corporation Deployment Policy (cont’d) Modified statements on a Deployment Policy: • First statement adds the filename value of Riba wherever it finds the filename value of Doggie. • Second statement changes any found value of Doggie to the new value of Riba. • Third statement deletes all metadata found under resource system. UEsDBBQAAAA………………. …………………..HIOAAAAAA== */*/services/xmlfirewall?Value=10001 change 11111 New in 3.7.1 Slide by David Shute IBM Software Group | WebSphere Software / DataPower © IBM Corporation Domain Import Policy � Add new/existing Deployment Policy to a configuration that you are importing New in 3.7.1 Slide by David Shute IBM Software Group | WebSphere Software / DataPower © IBM Corporation Summary/Recommendations � Various techniques were shown � Various levels of support for each � Be careful about creating port conflicts when bring in these packages to foreign domains! � Ideally version control and logical service granularity would be organized on a per domain level � xcfg and zip are functionally equivalent, but xcfg has almost no test or use coverage, so avoid xcfg � Export bundles are not designed to be executed with order dependencies: while the execution will be done in order, the processing of the import is async and inherently parallel � Export bundles are not designed to be staggered, meaning one includes the other. This will be inherently racy and is cumbersome. � ‘Questionable’ support for ‘hand-written’ CLI config files � Do not manually insert exec statements into a config file, the proper way to do this is by configuring an Include Configuration object. Only an Include Config object knows how to wait for the ethernet interface to be up on reboot. � Best solution for now seems to be export via XML, replace any localized values (hostname/IP, port, etc) with placeholders (%host%, %port%), checking into version control, then as part of deployment provide proper values from props files etc to replace placeholders, and deploy to target en