Friday, July 1, 2016

COMPLETE GOVERNANCE WITH GREG, APIM & BAM


Introduction


Service oriented architecture (SOA) is best way to develop softwares (services) to achieve any business use case. But SOA requires higher level of coordination and collaboration between lot of teams within the particular enterprise, from business teams to IT (information technology) teams and as well as among other teams and departments.This coordination and collaboration can be achieved by implementing a proper SOA governance model which deals with task, processes and people for defining and managing how services are created, supported and managed.


Governance is somewhat a political issue than a technical issue. While technology focuses on the interfaces, protocols and specifications, the business worry about method of serving customer. But both the technical and business emphasis on requirements to satisfy customers. Governance involves in all those aspects even those are separate efforts and processes. Governance conforms that everyone involved in those aspects are working together which are not contradicting with each other to ensure the financial side of the business is achieved as well as customer is satisfied.

What is governance



Exercising governance in SOA (Service oriented architecture) means implementing a process to ensure that everything is done according to protocols, defined guidelines, best practises, controls, management, rules, responsibilities and other related factors. Effective SOA governance must consider people, processes, technologies, deliverables, QOS (quality of services) in the entire lifecycle from identifying the business use case of a service to implementing, testing, delivering and up to reuse and until service is retired or no longer usable.


SOA governance consists two phases.
  1. Design time governance
    This includes the process from identifying a business use case to developing and implementing it as service.
  2. Run time governance
    This includes the process from delivering the service to end users to consume and enforcing policies to manage and control who can access the service and what they can do with it.


Reason for having two phases in governance is enforcing policies and SLAs from the service itself is expensive and unmanageable. This will be discussed more in runtime governance section.

Why Governance



There are many reasons why proper governance is needed not only for SOA but also everything we do. If there is not proper planning involved with the project, there is a higher chance that project end up in chaos.


Let’s imagine of a situations where there is no SOA governance involved with service development. Suppose X is working in ABC financial institute. And he is in loan branch and he has implemented a service where customers can view their current loan balance. Someone (Let’s say Y) in the saving branch implements another service which can be used to view customer’s current account balance. Account balance should show the loan balance too. And Y does not know that there is an existing service which he can use to get the loan balance. So Y too implements a new service which shows loan balance without using the existing service that X created.


Someone (Lets say Z) in another department in ABC company start using X loan balance service in his application. That applications start sending heavy traffic to the X’s loan service and crashes the server. Now X starts getting calls from people he does not know of, and complains that loan balance service is not working. Now X finds out that there are a lot of people other than Z in ABC company who use his loan balance service in their applications. Now X is in big trouble because his service is not working. This causes ABC company to lose their revenue and also it’s credibility to sustain their business.


This is a very simple scenario which could happen in any company or enterprise in the world right now. When there is a lack of effective SOA governance, repetition of services, unknown people using services, over usage of resource and services can happen. This could lead to major financial losses to a company.


But when there is proper SOA governance process is involved in the service lifecycle, it increases the possibility of achieving business goals and objectives.


Some reasons for having proper SOA governance in place


  1. Shows well structured responsibility management to empower people
  2. Can easily measure the effectiveness
  3. Well defined rules, protocols and policies to meet the business goals
  4. Avoid the repetitions  and reuse of existing service.
  5. Mechanism to ensure specifications are followed to the details.

Design Time Governance

What is Design Time Governance



SOA design time governance is the process of enforcing rules and protocols from identifying the business and implementing it as service. This is related the design time service cycle. As you can see in the following diagram illustrate the design time life cycle.





SOA governance starts with identifying the business use case. Most probably this is done by business analyst or somebody who has domain knowledge in business. They analyze the modern business trends and find out business opportunities. They bridge the business problems to technology solutions.


Next stage of SOA governance lifecycle implementing the business requirement as services. This is typically done by the software developers. They should follow SOA architecture and specifications to develop those services. They must ensure that defined protocols, guidelines and rules are followed to the point.


Then implemented service(s) should tested by developers, they should create unit test, functional test and etc… to complete the testing.


Next phase of design time life cycle is QA testing. This is the stage which services are tested for validations (whether developed services fulfill the business requirement), performance test (checks how the load is handled by the service, maximum load before it crashes)


Next stage of the service is sending on production for consume. This is a critical moment where runtime governance starts come in to picture and there some decisions has to be taken such as


  • who can invoke the service ?
  • what they can do with it ?
  • How long the service will be available for consumption ?
  • How many request will be served in a given moment ?


Once the service(s) is in production, it belongs to runtime governance which will be discussed it the runtime governance section.


Final state of service lifecycle is retiring service(s) where it has come to a point that current implementation of the service is no longer valid  for current business requirements. Retirement can happen due to new versions of same service is implemented.

Use of tools in Design time Governance



To enforce proper design time governance we must use tools designed for that purpose. WSO2 GREG is a specially designed tool to cater these requirement. We capture the meta data of the service as everything we discussed so far. Such as

Runtime Governance



Why do we have a run time governance phase in the SOA governance. Answer is simple. It is very inefficient and ineffective to build runtime governance capabilities to service implementation itself. Let’s us examine what are the requirement in runtime governance phase. So we may better understand why it requires to separate runtime and design time governance


  1. Access control
    1. Authentication  - (Who can use the service)
    2. Authorization  - (What they can do with it)
  2. Logging
  3. Enforcing policies
  4. Versioning
  5. Statistics and Monitoring
    1. Response time
    2. Success and failure rates
    3. Per user usage
    4. Per service usage
    5. etc..
  6. etc… (This list is open ended)



As show in the above image service implementation must be separated from policy enforcement (runtime governance). If not runtime governance requirements have to be implemented with service itself. Then it becomes a nightmare to manage both service and runtime governance requirements. So separation of service and policy enforcement is the most suitable way to achieve runtime governance.

Monitoring and Statistic



This is the one of the most critical requirement in SOA runtime governance. This gives the service provider ability to measure the effectiveness of the services provided.

Designtime and Runtime Governance Together



Please consider the following image.



As you can see I have aggregated both the design time governance and run time governance together in a single image. A strangest thing to notice here is that runtime governance is used in four stages of design time governance.


The reason for this is runtime governance should not be implemented when the service is in production. It should be tested and verified in from the service implementation state to production state. So the policy enforcing should be done as soon as implementation of the service started. If it is not done in those states of the design time life cycle, there will be a lot of complication when applying them on the production state. And ramifications will bring a nightmare to devOps to correct the issues in SLA policies. Even those policies must go through the testing and validating criterias.

Use case



Let’s imagine a use case where a company needs a complete governance process. Company X is financial company, who are mainly focused on giving it’s customers, a full financial solution from saving, loans, managing funds and brokering. This company has hundreds of employees from business analyst to software engineers to devops. They do various tasks in service life cycle. Business analysts are responsible for analysing business and proposing solutions. Software engineers are responsible for implementing the proposed business solutions as services. Devops are responsible for delivering the services in efficient and reliable way.


So let’s imagine a case where there is a need of a service to view the current account balance. Following activities will be done and questions will be answered to provide this view account balance functionality as a service.


  1. Identify the use case -
    1. Why it is needed ?
    2. What is the gap this service will be filling ?
    3. How this will benefit the company revenue ?
    4. Is this a urgent requirement ?
    5. Is there any other services identical to this which fulfils the requirement ?
  2. Implement the requirement as a service -
    1. What is the language used to implement the service ?
    2. Is there a web service documentation ?
    3. Who is responsible for developing the service ?
    4. What is the service name and the version ?
  3. Developer test -
    1. Does the service works without breaking anything ?
    2. Has checked with a code compatibility tool ?
    3. Is error handling properly done ?
  4. QA Testing -
    1. Does the implemented service fulfil the business requirement ?  (acceptance test)
    2. Does the service works smoothly ? (functional testing)
    3. Is seamless integration with other services possible ? (integration test)
    4. How does the service reacts to a high load ? (Load test)
  5. Deploy on public server for public usage
    1. Enforcing of rules and policies. This will be discussed later .
  6. Retiring the service
    1. Is there anyone using this service or has everyone migrated to new version ?


Now we have an idea what are the details we need to capture in every state of the service life cycle. So we need some kind tool to record all of this data. This is the place for WSO2 product stack to play a role in this situation.


GREG is for design-time governance and APIM for run-time governance.
Let’s see how we can use both WSO2 GREG and WSO2 APIM for the above use case.

GREG



WSO2 GREG is product specially designed to provide the right level of governance for SOA. You can find more information about this in following url.
You can download  and try WSO2 GREG from this location


APIM



WSO2 APIM is a complete solution for managing apis, routing traffic, specially for managing the runtime governance.
The documentation about the recent release is available here
Latest WSO2 APIM product can be downloaded from this url

Integrated Solution



Please look at the following diagram. The deployment of both GREG and APIM will be as follows according to what we discussed in the article.




As shown in left side of the image there is a GREG cluster which is responsible for Design time governance. And there is a clusters of API Managers which are responsible for runtime governance.


Let me explain what is rationale behind the image.
GREG cluster used as the tool for managing meta data related to design time governance. It will store meta data related schemas, wsdls, apis, owners, urls, policies etc… and will be responsible for migrating the metadata between different design time life cycle status.
As seen in the image there are separate clusters of APIM in each environment. Those are acting as the runtime governance enforcers. Those will help the enforce SLAs to service consumers.
BAM (DAS) will be used as the monitoring tool which will keep track of usage related data.

As middleware company, WSO2 has full stack of products to support both design and runtime governance effectively.


Wednesday, March 26, 2014

JAX-RS Tryit

If there is a better way to try your rest endpoints without writing a client, which will make  your life much easier. And even the clients who has subscribed to your restful services, can try them without problem, that would save a lot of time.
Now you have a solution. We WSO2 company, will be releasing WSO2 Application Server 5.3.0 at the end of April 2014, which includes this time saving feature.

Wednesday, February 19, 2014

WSO2 ESB 4.8.1 becomes the world's super fast ESB.

WSO2 ESB has become the leader of fastest opensource ESB in the world. Following graph and table tells the whole story. You can find more information about this performance testing from here.







Tuesday, January 7, 2014

WSO2 ESB supporting VFS transport with FTPS Protocol with File Encryption

WSO2 ESB supports vfs transports which is based on  the Apache Commons VFS implementation

WSO2 ESB supports many service level parameters in synapse configurations. Following are 3 service level parameters which you can use for protecting your files being transferred.

PASSIVE MODE

Passive mode is generally used in situations where the FTP server is not able to establish the data channel. One of the major reasons for this is network firewalls. While you may have a firewall rule which allows you to open up FTP channels to ftp.wso2.com, WSO2's servers may not have the power to open up the data channel back through your firewall. Passive mode solves this by opening up both types of channel from the client side.

IMPLICIT MODE

FTP over SSL (Implicit) Implicit security is a mechanism by which security is automatically turned on as soon as the FTP client makes a connection to an FTP server. In this case, the FTP server defines a specific port for the client (990, Can be change) to be used for secure connections.

And the other mode is explicit mode. Explicit security mechanism requires that the FTP client issues a specific command to the FTP server after establishing a connection to establish the secure (SSL) link. In explicit mode  the FTP client needs to send an explicit command ( i.e. "AUTH SSL" or "AUTH TLS") to FTP server to initiate a secure control connection.

PROT P

PROT P refers to the data transfers. Communication with the server is always encrypted if you use SSL/TLS. If PROT P isn't enforced, client could send PROT C and transfer files unencrypted. If PROT P is enforced, PROT C is rejected.


Using above 3 service level parameters you can establish a secure connection to transfer your files in FTP server using VFS protocol.

Following is a sample synapse configuration (proxy), which can be used to transfer files through WSO2 ESB.


<?xml version="1.0" encoding="UTF-8"?>
<proxy xmlns="http://ws.apache.org/ns/synapse"
       name="VFSSecureTransferProxy"
       transports="vfs"
       startOnLoad="true"
       trace="disable">
   <description/>
   <target>
      <inSequence>
         <log level="full"/>
         <drop/>
      </inSequence>
   </target>
   <parameter name="transport.PollInterval">1</parameter>
   <parameter name="transport.vfs.ActionAfterProcess">MOVE</parameter>
   <parameter name="transport.vfs.FileURI">vfs:ftps://{username}:{password}@{hostname/ip_address}:990/{source_filepath}?vfs.implicit=true&vfs.passive=true&vfs.protection=P</parameter>
   <parameter name="transport.vfs.MoveAfterProcess">vfs:ftps://{username}:{password}@{hostname/ip_address}:990/{destination_filepath}?vfs.implicit=true&vfs.passive=true&vfs.protection=P</parameter>
   <parameter name="transport.vfs.MoveAfterFailure">vfs:ftps://{username}:{password}@{hostname/ip_address}:990/{error_filepath}?vfs.implicit=true&vfs.passive=true&vfs.protection=P</parameter>
   <parameter name="transport.vfs.FileNamePattern">.*.xml</parameter>
   <parameter name="transport.vfs.ContentType">application/xml</parameter>
   <parameter name="transport.vfs.ActionAfterFailure">MOVE</parameter>
</proxy>






WSO2 ESB Removing full soap header using enrich mediator.

WSO2 ESB supports many mediators, and we can use them to achieve most of our use cases.

The following message template illustrates the structure of a SOAP Envelope:


<soap:Envelope   xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/">
  <soap:Header> <!-- optional -->
    <!-- header blocks go here... -->
  </soap:Header>
  <soap:Body>
    <!-- payload or Fault element goes here... -->
  </soap:Body>
</soap:Envelope>



Even though soap header is optional, you might receive the soap envelop with soap headers. And you might need to remove the soap header element including <soap:Header></soap:Header> element.

You can achieve this by using enrich mediator and payload factory mediator.
We first extract the body from the soap envelop and assign it to a property using enrich mediator. Then we create an empty soap envelop using payload factory. At last enrich mediator is again used to put the soap body to soap envelop from the property.

          <enrich>
            <source type="body" clone="true"/>
            <target type="property" property="ORIGINAL_BODY"/>
         </enrich>

         <payloadFactory media-type="xml">
            <format>
               <soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/">
                  <soapenv:Body/>
               </soapenv:Envelope>
            </format>
            <args/>
         </payloadFactory>

         <enrich>
            <source type="property" clone="true" property="ORIGINAL_BODY"/>
            <target type="body"/>
         </enrich>

Monday, November 25, 2013

WSO2 ESB append full body to a POST parameter

When you have to invoke a back end service which can take only on parameter and you need to send your full soap envelop body as single post parameter you can use script mediator and payload factory mediator as following. I have an api which calls to "http://localhost:9000/services/SimpleStockQuoteService"  and response is received by the "second_sequence".
In the second_sequence response is processed and assign it single parameter using script mediator and payload factory.

API


<api xmlns="http://ws.apache.org/ns/synapse" name="StockQuoteAPI2" context="/stockquote_api">
   <resource methods="GET" uri-template="/view/{symbol}">
      <inSequence>
         <payloadFactory>
            <format>
               <m0:getQuote xmlns:m0="http://services.samples">
                  <m0:request>
                     <m0:symbol>$1</m0:symbol>
                  </m0:request>
               </m0:getQuote>
            </format>
            <args>
               <arg expression="get-property('uri.var.symbol')"/>
            </args>
         </payloadFactory>
         <send receive="second_sequence">
            <endpoint>
               <address uri="http://localhost:9000/services/SimpleStockQuoteService" format="soap11"/>
            </endpoint>
         </send>
      </inSequence>
      <outSequence>
         <send/>
      </outSequence>
   </resource>
</api>
                     

SEQUENCE



<sequence xmlns="http://ws.apache.org/ns/synapse" name="second_sequence">
   <property name="DISABLE_CHUNKING" value="true" scope="axis2" type="STRING"/>
   <script language="js">
      <![CDATA[
                     var payload = mc.getEnvelope().getBody().getFirstElement().toString();
                     payload = encodeURI(payload);        
                     mc.setProperty("test",payload);
        ]]>
    </script>
      <payloadFactory>
         <format>
            <getSimpleQuote>
               <symbol>$1</symbol>
            </getSimpleQuote>
         </format>
         <args>
            <arg xmlns:ns="http://org.apache.synapse/xsd" expression="$ctx:test"/>
         </args>
      </payloadFactory>
      <log level="custom">
         <property xmlns:ns="http://org.apache.synapse/xsd" name="xxxxxxx....................." expression="$ctx:test"/>
      </log>
      <log level="full"/>
      <send>
         <endpoint>
            <address uri="http://localhost:9000/services/SimpleStockQuoteService" format="get"/>
         </endpoint>
      </send>
      <log level="full" category="DEBUG"/>
   </sequence>


You will find that in the sequence it sends the request as follows

GET /services/SimpleStockQuoteService/getSimpleQuote?symbol=%253cns:getQuoteResponse%2520xmlns:ns=%2522http://services.samples%2522%253e%253cns:return%2520xmlns:ax21=%2522http://services.samples/xsd%2522%2520xmlns:xsi=%2522http://www.w3.org/2001/XMLSchema-instance%2522%2520xsi:type=%2522ax21:GetQuoteResponse%2522%253e%253cax21:change%253e-2.878332384294565%253c/ax21:change%253e%253cax21:earnings%253e13.343243692880526%253c/ax21:earnings%253e%253cax21:high%253e83.90879243287203%253c/ax21:high%253e%253cax21:last%253e80.71927384911324%253c/ax21:last%253e%253cax21:lastTradeTimestamp%253eMon%2520Nov%252025%252012:10:46%2520PST%25202013%253c/ax21:lastTradeTimestamp%253e%253cax21:low%253e84.13935291865967%253c/ax21:low%253e%253cax21:marketCap%253e4.856632338341972E7%253c/ax21:marketCap%253e%253cax21:name%253eibm%2520Company%253c/ax21:name%253e%253cax21:open%253e84.41127937872128%253c/ax21:open%253e%253cax21:peRatio%253e-17.33795236794525%253c/ax21:peRatio%253e%253cax21:percentageChange%253e3.7341845696623412%253c/ax21:percentageChange%253e%253cax21:prevClose%253e-77.08061373503116%253c/ax21:prevClose%253e%253cax21:symbol%253eibm%253c/ax21:symbol%253e%253cax21:volume%253e9235%253c/ax21:volume%253e%253c/ns:return%253e%253c/ns:getQuoteResponse%253e






Monday, November 11, 2013

SSL handshake failed: SSL error: Key usage violation in certificate has been detected

SVN causing problems in ubuntu

When I was checking out a code in a svn repository which can only connect through https, started giving me errors as following.

SSL handshake failed: SSL error: Key usage violation in certificate has been detected.

This is caused by a bug in the version of libneon, fortunately this is fixed in version 0.29.3.  

To fix the problem, please follow the steps


  1. Uninstall the current libneon package: sudo apt-get remove libneon27
  2. Download the latest libneon package http://packages.debian.org/squeeze/libneon27
  3. Install the required libssl dependency: sudo apt-get install libssl0.9.8
  4. Install the downloaded libneon package: dpkg -i libneon27_0.29.3-3_amd64.deb
  5. Make the changes to symbolic links as described above: sudo mv /usr/lib/libneon-gnutls.so.27 /usr/lib/libneon-gnutls.so.27.old sudo ln -s /usr/lib/libneon.so.27 /usr/lib/libneon-gnutls.so.27