Manee's Tech Space: APIM

Showing posts with label APIM. Show all posts

Sunday, July 3, 2016

Wants to monitor outgoing or incoming messages ? Let's Enable WireLogs

Have you ever faced a situation where you have to monitor outgoing or incoming message of your WSO2 product in order to understand its flow of request/response transmission ?

If the answer is YES, we can simply use wirelogs in order to monior incoming and outgoing messages just do a simple changes in configurations. In here I'm considering WSO2 API Manager product to describe on how to enable wirelogs.

1. First go to <APIM_HOME>/repository/conf and open log4j.properties file.
2. In here you just simple need to uncomment the following line.

log4j.logger.org.apache.synapse.transport.http.wire=DEBUG

3. That's it now save the file and start API Manager server ( Go to <APIM_HOME>/bin and enter sh wso2server.sh )
4. Assuming an API has been already deployed, now go to Store ( https://<ip-address>:9443/store)
5. Go to API console of an already available API and tryout an available resource.
6. Then go to terminal of API Manager.

You can see the wirelogs were enabled and the message transmission are separated as follows.

Logs with '>>' are the outgoing transmissions from API Manager

Logs with '<<' are the incoming transmissions to API Manager

[2016-07-01 22:07:33,493] DEBUG - wire >> "OPTIONS /pizzashack/1.0.0/menu HTTP/1.1[\r][\n]"
[2016-07-01 22:07:33,493] DEBUG - wire >> "Host: 172.16.2.23:8243[\r][\n]"
[2016-07-01 22:07:33,493] DEBUG - wire >> "Connection: keep-alive[\r][\n]"
[2016-07-01 22:07:33,494] DEBUG - wire >> "Access-Control-Request-Method: GET[\r][\n]"
[2016-07-01 22:07:33,494] DEBUG - wire >> "Origin: https://172.16.2.23:9443[\r][\n]"
[2016-07-01 22:07:33,494] DEBUG - wire >> "User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.87 Safari/537.36[\r][\n]"
[2016-07-01 22:07:33,494] DEBUG - wire >> "Access-Control-Request-Headers: accept, authorization[\r][\n]"
[2016-07-01 22:07:33,494] DEBUG - wire >> "Accept: */*[\r][\n]"
[2016-07-01 22:07:33,494] DEBUG - wire >> "Referer: https://172.16.2.23:9443/store/apis/info?name=PizzaShackAPI&version=1.0.0&provider=admin&tenant=carbon.super[\r][\n]"
[2016-07-01 22:07:33,494] DEBUG - wire >> "Accept-Encoding: gzip, deflate, sdch[\r][\n]"
[2016-07-01 22:07:33,494] DEBUG - wire >> "Accept-Language: en-US,en;q=0.8[\r][\n]"
[2016-07-01 22:07:33,494] DEBUG - wire >> "[\r][\n]"
[2016-07-01 22:07:33,565] DEBUG - wire << "HTTP/1.1 200 OK[\r][\n]"
[2016-07-01 22:07:33,566] DEBUG - wire << "Origin: https://172.16.2.23:9443[\r][\n]"
[2016-07-01 22:07:33,566] DEBUG - wire << "Accept: */*[\r][\n]"
[2016-07-01 22:07:33,566] DEBUG - wire << "Access-Control-Request-Method: GET[\r][\n]"
[2016-07-01 22:07:33,566] DEBUG - wire << "Access-Control-Allow-Origin: *[\r][\n]"
[2016-07-01 22:07:33,566] DEBUG - wire << "Access-Control-Allow-Methods: GET[\r][\n]"
[2016-07-01 22:07:33,566] DEBUG - wire << "Access-Control-Request-Headers: accept, authorization[\r][\n]"
[2016-07-01 22:07:33,566] DEBUG - wire << "Referer: https://172.16.2.23:9443/store/apis/info?name=PizzaShackAPI&version=1.0.0&provider=admin&tenant=carbon.super[\r][\n]"
[2016-07-01 22:07:33,566] DEBUG - wire << "Host: 172.16.2.23:8243[\r][\n]"
[2016-07-01 22:07:33,566] DEBUG - wire << "Accept-Encoding: gzip, deflate, sdch[\r][\n]"
[2016-07-01 22:07:33,566] DEBUG - wire << "Accept-Language: en-US,en;q=0.8[\r][\n]"
[2016-07-01 22:07:33,566] DEBUG - wire << "Access-Control-Allow-Headers: authorization,Access-Control-Allow-Origin,Content-Type,SOAPAction[\r][\n]"
[2016-07-01 22:07:33,566] DEBUG - wire << "Date: Sun, 03 Jul 2016 16:37:33 GMT[\r][\n]"
[2016-07-01 22:07:33,566] DEBUG - wire << "Transfer-Encoding: chunked[\r][\n]"
[2016-07-01 22:07:33,566] DEBUG - wire << "Connection: keep-alive[\r][\n]"
[2016-07-01 22:07:33,566] DEBUG - wire << "[\r][\n]"
[2016-07-01 22:07:33,566] DEBUG - wire << "0[\r][\n]"
[2016-07-01 22:07:33,566] DEBUG - wire << "[\r][\n]"
[2016-07-01 22:07:33,570] DEBUG - wire >> "GET /pizzashack/1.0.0/menu HTTP/1.1[\r][\n]"
[2016-07-01 22:07:33,570] DEBUG - wire >> "Host: 172.16.2.23:8243[\r][\n]"
[2016-07-01 22:07:33,570] DEBUG - wire >> "Connection: keep-alive[\r][\n]"
[2016-07-01 22:07:33,570] DEBUG - wire >> "Accept: application/json[\r][\n]"
[2016-07-01 22:07:33,571] DEBUG - wire >> "Origin: https://172.16.2.23:9443[\r][\n]"
[2016-07-01 22:07:33,571] DEBUG - wire >> "User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.87 Safari/537.36[\r][\n]"
[2016-07-01 22:07:33,571] DEBUG - wire >> "Authorization: Bearer 3de15404-ffdb-3427-98da-65e6e43aaeaa[\r][\n]"
[2016-07-01 22:07:33,571] DEBUG - wire >> "Referer: https://172.16.2.23:9443/store/apis/info?name=PizzaShackAPI&version=1.0.0&provider=admin&tenant=carbon.super[\r][\n]"
[2016-07-01 22:07:33,571] DEBUG - wire >> "Accept-Encoding: gzip, deflate, sdch[\r][\n]"
[2016-07-01 22:07:33,571] DEBUG - wire >> "Accept-Language: en-US,en;q=0.8[\r][\n]"
[2016-07-01 22:07:33,571] DEBUG - wire >> "[\r][\n]"

Friday, June 3, 2016

Setup WSO2 API Manager Analytics with WSO2 API Manager 2.0 using RDBMS

In this blog post I'll explain on how to configure RDBMS to publish APIM analytics using APIM analytics 2.0.0.

The purpose of having RDBMS is to fetch and store summarized data after the analyzing process. API Manager used this data to display on APIM side using dashboards.

Since the APIM 2.0.0, RDBMS use as the recommended way to publish statistics for API Manager. Hence, I will explain step by step configuration with RDBMS in order to view statistics in Publisher and Store through this blog post.

Steps to configure,

1. First download the WSO2 API Manager Analytics 2.0.0 release pack and unzip it.

( Download )

2. Go to carbon.xml ([APIM_ANALYTICS_HOME]/repository/conf/carbon.xml) and set port offset as 1 (default offset for APIM Analytics)

<Ports>
<!-- Ports offset. This entry will set the value of the ports defined below to
the define value + Offset.
e.g. Offset=2 and HTTPS port=9443 will set the effective HTTPS port to 9445
-->
<Offset>1</Offset>

Note - This is only necessary if both API Manager 2.0.0 and APIM Analytics servers run in a same machine.

3. Now add the data source for Statistics DB in stats-datasources.xml ([APIM_ANALYTICS_HOME]/repository/conf/datasources/stats-datasources./xml) according to the preferred RDBMS. You can use any RDBMS such as h2, mysql, oracle, postgres and etc and here I choose mysql to use in this blog post.

<datasource>
  <name>WSO2AM_STATS_DB</name>
  <description>The datasource used for setting statistics to API Manager</description>
  <jndiConfig>
    <name>jdbc/WSO2AM_STATS_DB</name>
    </jndiConfig>
  <definition type="RDBMS">
    <configuration>
      <url>jdbc:mysql://localhost:3306/statdb?autoReconnect=true&amp;relaxAutoCommit=true</url>
      <username>maneesha</username>
      <password>password</password>
      <driverClassName>com.mysql.jdbc.Driver</driverClassName>
      <maxActive>50</maxActive>
      <maxWait>60000</maxWait>
      <testOnBorrow>true</testOnBorrow>
      <validationQuery>SELECT 1</validationQuery>
      <validationInterval>30000</validationInterval>
      </configuration>
    </definition>
</datasource>

Give the correct hostname and name of the db in <url> (in this case, localhost and statdb respectively), username and password for the database and drive class name.

4. WSO2 analytics server automatically create the table structure for statistics database at the server start up using ‘-Dsetup’.

5. Copy the related database driver into <APIM_ANALYTICS_HOME>/repository/components/lib directory.

If you use mysql - Download
If you use oracle 12c - Download
If you use Mssql - Download

6. Start the Analytics server

7. Download the WSO2 API Manager 2.0.0 pack and unzip it ( Download )

8. Open api-manager.xml ([APIM_HOME]/repository/conf/api-manager.xml ) and enables the Analytics. The configuration should be look like this. (by default the value set as false)

<Analytics>
        <!-- Enable Analytics for API Manager -->
        <Enabled>true</Enabled>

9. Then configure Server URL of the analytics server used to collect statistics. The define format is ' protocol://hostname:port/'. Although admin credentials to login to the remote DAS server has to be configured like below.

<DASServerURL>{tcp://localhost:7612}</DASServerURL>
<DASUsername>admin</DASUsername>
<DASPassword>admin</DASPassword>

Assuming Analytics server in the same machine as the API Manager 2.0, the hostname I used here is 'localhost'. Change according to the hostname of remote location if the Analytics server runs on a different instance.

By default, the server port is adjusted with offset '1'. If the Analytics server has a different port offset ( check {APIM_ANALYTICS_HOME}/repository/conf/carbon.xml for the offset ), change the port in <DASServerURL> accordingly. As an example if the Analytics server has the port offset of 3, <DASServerURL> should be {tcp://localhost:7614}.

10. For your information, API Manager 2.0 enables RDBMS configuration to proceed with statistics, by default. To enable publishing using RDBMS, <StatsProviderImpl> should be uncommented (By default, it's not in as a comment. So this step can be omitted)

<!-- For APIM implemented Statistic client for DAS REST API -->
        <!--StatsProviderImpl>org.wso2.carbon.apimgt.usage.client.impl.APIUsageStatisticsRestClientImpl</StatsProviderImpl-->
        <!-- For APIM implemented Statistic client for RDBMS -->
        <StatsProviderImpl>org.wso2.carbon.apimgt.usage.client.impl.APIUsageStatisticsRdbmsClientImpl</StatsProviderImpl>

11. The next step is to configure the statistics database in API Manager side. Add the data source for Statistics DB which used to configure in Analytics by opening master-datasources.xml ([APIM_HOME]/repository/conf/datasources/master-datasources./xml)

<datasource>
  <name>WSO2AM_STATS_DB</name>
  <description>The datasource used for setting statistics to API Manager</description>
  <jndiConfig>
    <name>jdbc/WSO2AM_STATS_DB</name>
    </jndiConfig>
  <definition type="RDBMS">
    <configuration>
      <url>jdbc:mysql://localhost:3306/statdb?autoReconnect=true&amp;relaxAutoCommit=true</url>
      <username>maneesha</username>
      <password>password</password>
      <driverClassName>com.mysql.jdbc.Driver</driverClassName>
      <maxActive>50</maxActive>
      <maxWait>60000</maxWait>
      <testOnBorrow>true</testOnBorrow>
      <validationQuery>SELECT 1</validationQuery>
      <validationInterval>30000</validationInterval>
      </configuration>
    </definition>
</datasource>

12. Copy the related database driver into <APIM_HOME>/repository/components/lib directory as well.

13. Start the API Manager server.

Go to statistics in publisher and the screen should looks like this with a message of 'Data Publishing Enabled. Generate some traffic to see statistics.'

To view statistics, you have to create at least one API and invoke it in order to get some traffic to display in graphs.

Sunday, May 8, 2016

Setup WSO2 API Manager Analytics with WSO2 API Manager 2.0 using REST Client

Please Note - Statistics publishing using REST Client was deprecated from APIM 2.0.0. Please refer this to continue.

In this blog post I will explain how to configure WSO2 API Manager Analytics 2.0.0 with WSO2 API Manager 2.0 to publish and view statistics. Before going further into the topic, I thought to give a brief summary about the role of WSO2 API Manager Analytics 2.0.0 in here

WSO2 API manager embedded with the ability to view statistics of the operations carried out such as usage comparison, monitoring Throttled Out Requests, API last access time and so on. To view so, the user has to configure an analytics server with API Manager and it allows to view statistics based on the given criteria. Until WSO2 API Manager 2.0.0, the recommended analytics server to view statistics was WSO2 DAS ( Data Analytics Server ) which is a high performing enterprise data analytics platform. Before that WSO2 BAM (Business Activity Monitor) used to collect and analyze runtime statistics from the API Manager. Based on the WSO2 DAS, with the vision of having a separate but custom analytics package including new features that will perform all the analytics for API Manager, WSO2 API Manager Analytics has been introduced. WSO2 API Manager analytics fuses batch and real-time analytics with predictive analytics and generate alerts when an abnormal situation occurs via machine learning.

Hope now you have a sound knowledge on what API Manager analytics is all about. So let's starts with configuration.

Steps to configure,

1. First download the WSO2 API Manager Analytics 2.0.0 release pack and unzip it.

( Download )

2. Start the Analytics server (By default the port offset was given as 1 in carbon.xml)

3. Go to Management Console of Analytics Server and logged in as administrator (Username- admin, Password- admin).

4. Go to Manager -> Carbon Applications. List and delete the existing org.wso2.carbon.analytics.apim carbon app.

5. Browse Rest Client car app (org_wso2_carbon_analytics_apim_REST-1.0.0.car) from [APIM_ANALYTICS_HOME]/statistics and upload.

That's it from APIM Analytics side. Now see how to configure API Manager to finalize the configurations.

6. Download the WSO2 API Manager 2.0.0 pack and unzip it ( Download )

7. Open api-manager.xml ([APIM_HOME]/repository/conf/api-manager.xml ) and enables the Analytics. The configuration should be look like this. ( by default the values set as false )

<Analytics> 
        <!-- Enable Analytics for API Manager --> 
        <Enabled>true</Enabled>

8. Then configure Server URL of the analytics server used to collect statistics. The define format is ' protocol://hostname:port/'. Although admin credentials to login to the remote DAS server has to be configured like below.

<DASServerURL>{tcp://localhost:7612}</DASServerURL> 
<DASUsername>admin</DASUsername> 
<DASPassword>admin</DASPassword>

Assuming Analytics server in the same machine as the API Manager 2.0, the hostname I used here is 'localhost'. Change according to the hostname of remote location if the Analtics server run on different instance. By default the server port is adjusted with offset '1'. If the Analytics server has a different port offset ( check [APIM_ANALYTICS_HOME]/repository/conf/carbon.xml for the offset ), change the port in <DASServerURL> accordingly. As an example if the Analytics server has the port offset of 3, <DASServerURL> should be {tcp://localhost:7614}.

Now we have to choose between 2 clients to fetch and publish statistics.

The RDBMS client which fetches data from RDBMS and publish.
The REST client which directly fetches data from Analytics server.

I chose REST client to publish data in this tutorial and will explain how to configure the data fetching using RDBMS in next blog post.

For your information, API Manager 2.0 enables RDBMS configuration to proceed with statistics, by default.

9. To enable publishing using REST Client, <StatsProviderImpl> should be uncommented (By default, it's in as a comment) and comment <StatsProviderImpl> for RDBMS

<!-- For APIM implemented Statistic client for DAS REST API -->
        <StatsProviderImpl>org.wso2.carbon.apimgt.usage.client.impl.APIUsageStatisticsRestClientImpl</StatsProviderImpl>
        <!-- For APIM implemented Statistic client for RDBMS -->
        <!--StatsProviderImpl>org.wso2.carbon.apimgt.usage.client.impl.APIUsageStatisticsRdbmsClientImpl</StatsProviderImpl-->

10. Then the REST API url should be configured with hostname and port along with the credentials to access,

<DASRestApiURL>https://localhost:9444</DASRestApiURL> 
<DASRestApiUsername>admin</DASRestApiUsername> 
<DASRestApiPassword>admin</DASRestApiPassword>

As mentioned before, the port associate with the default offset of 1 for WSO2 APIM analytics 1.0.0.

11. Now Save api-manager.xml and start the API Manager 2.0 server.

That's it. Open publisher in a browser ( https://<ip-address>:<port>/publisher). Go to Statistics and select API Usage as an example. The screen should looks like this with a message of 'Data Publishing Enabled. Generate some traffic to see statistics.'

Just create few APIs and try to invoke them in order to get some traffic to generate statistics on graph. So you can see the statistics like this.