Integration Server

Post Reply
dsk35
Posts: 2
Joined: 08 Apr 2009, 13:49

Hi
You mentioned an integration server that you are working on.
I wondered how you were progressing and whether that would help in importing a number of different physical and logical configurations.

Thanks

Dan
jason.powell
Posts: 32
Joined: 04 Feb 2009, 15:01

Dan,

at this time, we don't have a date set for the release of the Integration Server, but it is one of our high priority activities. As soon as it is ready for community testing, we will post a notification on the forum.

Regarding your being able to import logical and physical configurations, the key issue will be the source or format of the information to be imported (e.g. Excel spreadsheet, XML file, commercial modelling tool) as the initial release of the Integration Tool will be focused on information captured in an XML-based format.

Regards,

Jason
(Essential Project Team)
Essential Project Team
charmes
Posts: 5
Joined: 01 Apr 2009, 16:26

Until the Integration Server becomes avaialable, would it be possible for you to provide a target XML format/schema that we could use in a manual process to one-time import existing instance information?

Not looking for anything elaborate, just something to help jump-start our EA implementation for those of us that have a large amount of existing architecture information.

Thanks.
User avatar
jonathan.carter
Posts: 1087
Joined: 04 Feb 2009, 15:44

We are going to provide a 'pre-alpha' version of the Essential Integration Server for you to use. There are still some manual steps to go through but it works nicely in terms of importing from external sources (and even synchronising with them on an on-going basis). We're working on making this more "one-click" but I think you may find it useful in its current form - and I'd be more than happy to help you with any problems you find. I'll get this ready over the next day or so.

The idea behind the integration server is that you write some XSL to do the 'integration' between your existing information source and Essential. This is, naturally, a classic integration problem where you are mapping and transforming elements in one format to elements in another - and this can become complex depending on what you need to import / synchronise. However, we have a 'generic' XSL that imports other Essential Architecture Manager repositories into your current one. The XML used is the XML that is published to the Essential Viewer component, and I've posted an XSD (basically a cut-down version of the Protege Experimental XML) here for you to look at and use.

What I'd recommend is that you work towards transforming your source information into the form described by the XSD and then we'll be able to run the 'generic' Essential import XSL to bring your instance information into your Essential repository. Have a look at the reportXML.xml file in the /WebContent folder of the Essential Viewer web app for an example of how this represents your information.

To do this mapping, the main step is to identify the Essential Meta Classes that you will be mapping to. It's often easiest to import just the 'definitions' of the instances and then 'wire up' the relationships between these in Protege. What I mean by this is that you are creating an import that says things like:

Business Process, 'My Business Process' has a unique ID of 'instance_001'.
<simple_instance>
<name>instance_001</name>
<type>Business_Process</type>
<own_slot_value>
<slot_reference>name</slot_reference>
<value value_type="string">My Business Process</value>
</own_slot_value>
</simple_instance>
It is completely possible to create / import all the required relationships as part of the import process but that can get quite complex as any relationship slots use the unique name of the instance being linked to, which makes your transformations a bit complicated - but entirely possible.

We can discuss this further as required.

Have a look at the XSD and the XML and how your existing instance information maps to the Essential Meta Model and I'll get the 'pre-alpha' version of the Integration Server ready for download and with further instructions.

Hope this helps

Jonathan
Essential Project Team
User avatar
jonathan.carter
Posts: 1087
Joined: 04 Feb 2009, 15:44

I've uploaded a pre-alpha version of the Integration Server to our community share area.

Although this still has a few manual steps that we are working on automating, it performs the imports and on-going synchronisation correctly and I've been using this in anger for some months.
New developments to this include the ability to import all the instances from a published Essential Architecture Manager repository (published to the Essential Viewer) out-of-the-box.
Even in its current form, it will get all your existing information into Essential Architecture Manager.

If you can transform your current architecture information to the schema I mentioned in my last post, this out-of-the-box import will bring them all in. There's a very small demo provided in this distribution to show how it all works and so that you can see the expected behaviour.

As I described, doing this kind of import is something of a classic integration problem as we have to perform a semantic mapping from your source information to the Essential Meta Model, then transform the data as required to make the mapping work and as you can imagine, this often ends up with a bespoke integration / import for every 'external' data source.
The integration server is XSL-based but uses the Protege API via scripting (so, no compiling etc.) to perform the import and to ensure the referential integrity of your target Essential Architecture Manager repository. The XSL writes these API scripts and to make this as easy as possible, uses a set of 'standard functions' that I've put together that take care of all the tricky manipulation of the API.
Where I'm going with this is that you will be able to see from the 'generic' Essential importer XSL what sort of things need to be done and from that you can (if you need to) easily write your own import script. This may save you doing the transformation into the Essential Report Schema XML.

Let me know if you would like further help/details about doing that.

Installation instructions for the Integration Server and a user guide are included on the site.

Let me know how you get on, particularly if you have any trouble

Jonathan
Essential Project Team
charmes
Posts: 5
Joined: 01 Apr 2009, 16:26

Thanks for supplying the Integration Server. I have deployed it and have successfully imported over a thousand Hardware_Instance nodes. I continue to refine my source XML to bring additional information and relationships across.

I did notice one issue: If there are multiple occurrences of the same simple_instance node in the reportXML.xml, the generated script file errors out because it contains multiple instances of the string "Hardware_Instance" in the command as follows:
theInstance = getEssentialInstanceContains("Hardware_Instance Hardware_Instance Hardware_Instance", "Server_86D69E449C8A46468833CBD69CEF9D9F", "External Essential Repository", "MYSERVER")

The "Hardware_Instance" string appears to be repeated for each duplicate simple_instance node in the XML. inserted seems to be
charmes
Posts: 5
Joined: 01 Apr 2009, 16:26

Additional issue: script generator places quotes around all values regardless of value_type contained in source XML. This results in strings where integers were indicated. I have found that if I edit the script file and remove the quotes, the values are imported as integers and I do not get "red" boxes around them in the display.

For example:

aSlotName = "processor_quantity"
theInstance.setOwnSlotValue(kb.getSlot(aSlotName), "2")

should be:

aSlotName = "processor_quantity"
theInstance.setOwnSlotValue(kb.getSlot(aSlotName), 2)
User avatar
jonathan.carter
Posts: 1087
Joined: 04 Feb 2009, 15:44

Hi,

Great to hear that you're finding the Integration Server useful and thanks for raising this issue. A lot of the early use of the server was for just this purpose - importing physical node information.

By the way, as you might notice from the sample repository, physical servers are captured using the Technology_Node class - this is what the reports in Viewer expect. We did (in an earlier incarnation of the Meta Model) also capture the server as a Hardware_Instance but this became a bit of a problem, so we typically use this to capture information about interesting additional hardware, such as attached storage devices etc. that are attached to a node, but don't in themselves constitute a server as such. I realise that you've been working on this for some time before we delivered the sample repository.

I have a couple of questions that I'd like to clarify to get a better understanding of what's going on.

It sounds as if you are transforming your source information to the Essential Architecture Manager repository import schema, as used by reportXML.xml. Is this correct?

The schema might not be enforcing this, but there should never be duplicate simple_instance elements in the XML. The name, i.e. simple_instance/name must be unique across the entire XML document.
What is the reason that you have duplicate Hardware Instances in the reportXML.xml? I think this might be at the root of the problem.

Thanks.

Also, thanks for the update! Just got it as I was submitting the above.
OK, that is a problem that I will resolve in the XSL. Thanks for spotting that!

Jonathan
Essential Project Team
charmes
Posts: 5
Joined: 01 Apr 2009, 16:26

Yes, I have taken an extract from our existing tool and converted it to the XML format used by reportXML.xml.

Duplicates in the XML were certainly caused by duplicate information in the original source extract. I didn't realize we had duplicates in the extract until I received the errors during the script processing and investigated. While I agree this is the root issue, perhaps you may want to consider generating multiple insert/update entries when you encounter duplicates? It would provide the flexibility to use the XML import as a type of batch-entry mechanism to perform bulk updates without the user having to assemble all updates for a given instance into a singular definition. Since your script processing performs automatic create/update processing, it lends itself to this type of work.

Thanks for the information regarding the Technology_node. I will make the appropriate change. We are in the early prototype phase, learning as we go.

So far, I am very impressed with this Essential Architecture tool and what we can accomplish. Thanks for all your work and continued support.
User avatar
jonathan.carter
Posts: 1087
Joined: 04 Feb 2009, 15:44

I did some tests today and replicated the problem. I agree that it's better to have robust code that will handle duplicates.

I've posted an updated version of the Essential Integration Server to the Share area - but all you really need is the updated import XSL, which is also up there as a separate download.

Unzip that and drop it in [quote]<Tomcat>/EssentialIntegrationServer/IntegrationScripts/XSL

This update handles duplicate entries but I also noticed that if you have 2 instances with the same ID (i.e. /simple_instance/name) but different type values, then that was also causing trouble. Again, this is a case of bad source data but the import now takes the 'type' from the FIRST occurrence of the duplicate but will, as you suggested, generate duplicate statements which will first do a create, then updates. However, the script will run successfully.

I've also fixed the bugs around setting Integer, Float and Boolean slot values.

Really pleased to hear that you are finding all this useful - and thanks for pointing out these issues.

Jonathan
Essential Project Team
Eric.Feron
Posts: 3
Joined: 17 Nov 2009, 15:47

I'd like to import new "Technology_Product" instances linked with their corresponding "Supplier" instance.
For instance, if I want to create a product "Wireshark" linked to "Wireshark.org", I generated a reportXML.xml file containing :
<simple_instance>
<name>Wireshark.org</name>
<type>Supplier</type>
<\simple_instance>
and an instance "test_class233" has been created in "Supplier" via the Integration Server.
Then I create another reportXML.xml with :
<simple_instance>
<name>Wireshark</name>
<type>Technology_Product</type>
<own_slot_value>
<slot_reference>name</slot_reference>
<value value_type="string">Wireshark</value>
</own_slot_value>
<own_slot_value>
<slot_reference>supplier_technology_product</slot_reference>
<value value_type="simple_instance">test_class233</value>
</own_slot_value>
</simple_instance>
, but in the instance browser of Protege, the "Product Supplier" is not "Wireshark.org".
What's wrong in my last xml ? The value or the value_type ?
Thanks in advance.
User avatar
jonathan.carter
Posts: 1087
Joined: 04 Feb 2009, 15:44

Hi Eric,

The problem here is that in the reportXML.xml format, the <name> tag is the instance ID, e.g. test_class233, not the actual name, Wireshark.org.

The schema of the <simple_instance> tag is roughly:

Code: Select all

<simple_instance>
   <name>The INSTANCE ID - Protege's internal name for the instance</name>
   <type> The Class that this is an instance of </type>

   <own_slot_value> Every field on Protege forms are in these
      <slot_reference>name</slot_reference>
      <value value_type="string">The value of the name field on the Protege form</value>
   <own_slot_value>

   ...
</simple_instance>
So, this means that the <name> tag is what Protege uses to store the instance ID - the internal name if you will. Each of the fields that you see on the Protege forms are stored in the reportXML.xml in the <own_slot_value> tags, which is always a pair of the <slot_reference> which is the name of the field, e.g. in your case "name", and a <value> tag, which is the actual value of the field for this instance.

Hope this helps.
Sounds like you are using the "default" transformation in the Integration Server - that is converting your source to the reportXML.xml form before running the import. If this is proving a bit difficult, it is possible to define your own transforms in the Integration Server using XSLT. This XSLT is used to write the Python script that you run to do the import and there are some helper functions for this Python that you can use for creating instances in a very simple manner in your XSLT.
Sometimes this can be simpler than mapping directly to the reportXML.xml format.

Please don't hesitate to let me know if you would like to know more about this sort of thing.

Jonathan
Essential Project Team
Eric.Feron
Posts: 3
Joined: 17 Nov 2009, 15:47

Indeed, an example of XSL script, with corresponding XML files, could be usefull to better understand this second way of import.
Best regards,
Eric.
User avatar
jonathan.carter
Posts: 1087
Joined: 04 Feb 2009, 15:44

I've posted some examples in the Share Area.

The main principle of the XSL is that rather than transform the source to an XML target, the XSL generates a Python import script, using a set of helper functions that we have developed to manage the update/creation and synchronisation of instances as appropriate.

As a simple introduction to how this works, have a look in the 'IntegrationScripts/XSL' folder of the Essential Integration Server webapp for the XSL that drives the "default" Essential->Essential import that you are using - 'importEssentialInstances.xsl'. You will notice in the 'IntegrationScripts/' folder a file called 'standardFunctions.txt'. This is the library of import helper functions that I've put together.

You'll notice that in 'importEssentialInstances.xsl' it uses the function getEssentialInstanceContains() to find instances by name (the own_slot_value/slot_reference/value="name"). There are other 'finder' functions in there.

The examples that I've posted are based on an import from an external repository of physical technology information about server nodes and their operating systems. This gives a slightly more complex example than the 'importEssentialInstances.xsl' as it performs the transformation from the source XML to the import script. This can be useful where an element from the import needs to be split into separate instances in Essential and related to each other. e.g. the relationships between the nodes and the technology instances deployed on them.

At the moment, the Integration Server is not as slick as we would like it to be (I'm about to start looking at that). To use the examples, or to start working on your own imports, put your XSL file in the 'IntegrationScripts/XSL' folder and your XML source in the 'IntegrationScripts/XML' folder of the Essential Integration Server.
The integration server is invoked via an HTTP request, so you'll need to setup the following sort of request, e.g. in your browser address field, or create a new link on the Integration Server home page (index.html), which can be used to provide a set of different imports.

Code: Select all

http://<host>:8080/EssentialIntegrationServer/RepositoryImportServicexslFile=importEssentialInstances.xsl&
scriptFile=essentialImport&
externalRepository=XML/reportXML.xml&
externalRepositoryName=External%20Essential%20Repository&
externalRepositoryDesc=External%20Essential%20Repository%20Source
The fields are:
  • xslFile: Your XSL file. You do not need the path, just the filename.
  • scriptFile: the prefix of the script file that will be created to run in the Protege Script Tab
  • externalRepository: the path to the XML source file, from the root of the "EssentialIntegrationServer" folder. If you've put it in the XML folder, then use: XML/<source.xml>
  • externalRepositoryName: the name of the source that you are importing from. This is used for synchronisation and updates.
  • externalRepositoryDesc: a description of the source repository.
Note, in 'importEssentialInstances.xsl' the call:
<xsl:text>defineExternalRepository("</xsl:text><xsl:value-of select="$ExternalRepository"></xsl:value-of>...
that is used to setup the external repository definition in Essential, enabling us to manage/synchronise imports from multiple external sources at the same time.

I'd be more than happy to help with getting you started with your import if you think this kind of approach is of interest. Having said that, if your current approach of performing a transform to the Essential XML schema is working for you, there is no need to change approach.

Jonathan
Essential Project Team
User avatar
jonathan.carter
Posts: 1087
Joined: 04 Feb 2009, 15:44

Just a quick post to let you know that we've just released the Essential Integration Tab which replaces the Integration Server. It uses an evolution of the components of the integration server but provides it all in a much easier-to-use package.

Download it from the Community/Share area.

Jonathan
Essential Project Team
Post Reply