II.) Machine Learning and the Life cycle of Prediction

The fundamental principle of machine learning is to Predict.

Then how will I Predict?dataprediction

Lets take a very simple example to Predict

predictionexample

  • First, we will look in to previous data and find if there is any common pattern and apply that to predict.
  • So here the pattern is simple ,it’s  Y = 1.5 * X.We can also say Y= m*x ,where m=1.5.

Here Y is called as Dependent Variable and X is Independent Variable

But in real time it won’t be such simple.We will look at such small sample dataset

dataset1

These are the Profits for some startups in a particular state.We need to Predict the profits for some-other startupsdataprediction2

We will Follow the same procedure.

  • Look in to previous data where we already know Profits
  • Look for any Patterns . But we don’t find any such as it is not straight forward.Here Profit is Dependent on all the other variable.
    (R&D Spend,Administration,Marketing Spend,State)-Independent Variables.

So we can assume it as :

Profit = A1*(R&D Spend)+A2*(Administration)+A3*(Marketing Spend)+A4*(State)

There will still some more factors.But will just ignore those.

In Previous example we were easily able to eye ball the sequence.But here it is difficult. So we need some kind of calculator who will help me to find out the pattern!!We will use something called Algorithm who will help me to compute calculations.We will dump the data to algorithm and it will help us to find the pattern i.e.., A1,A2,A3,A4

Next is to validate/Test

Ideally we use 70% data for algorithm and keep 30% data aside for testing.Then we validate and calculate the average error .
Predict the Range

Once the average error is predicted we will predict the profit value+/-

machinelearning_prediction

High level Life cycle of Prediction with ML

There are lot more factors involved in real time but this is the general concept with prediction.We will look more about this in future blogs.

Previous Blog – S1

 

I.) Importance of Data and Intro to Machine Learning and AI

Data

Data is everywhere.For every decision we make we use data knowingly or unknowingly.Let it be going to movie,restaurant,company etc.,data here can be reviews,from your past experience etc., data can be from any source even that is stored in your memory from past experience.

Data can be broadly divided in to 3 branches

data_branch

Our main focus is on Data science.As part of Data-science, data-analyst’s will fetch data from MIS such as data warehouses etc and using business intelligence they make predictions.

Predictions are done by experience from Previous data.Predictive Analytics is important to make Business decisions.

predictive analytics

Talking about Human intelligence,the things which we can do:

neural networks

Our brain has the power to connect sequence .If we can mimic this in software where we can interpret sequences its called Recurrent neural network.

Understanding of language is nothing but Natural Language Processing. If we mimic this in a software that is nothing but chatbot. Most of the current chatbots are trained to look for keywords and accordingly respond .They can’t interpret the sequence.

Currently lot of research is going on ,how to apply Recurrent Neural network and Natural Language Processing for a chatbot, which is very complicated.
Below is the funny example of real-time chatbot

indigo_bot


Here chatbot didn’t understand Sarcasm.It just looked for keywords.Lot of research is being done to understand human emotions.
We will learn more about this in upcoming blogs.

Next Blog – S2

Useful apis and script to start BPD in IBM BPM

When a bpd is created In the overview instance name, to get instanceID

“Customer Process:” + tw.system.process.instanceId

To start a bpd using service

tw.local.inputValues=new tw.object.Map();
tw.local.inputValues.put('idjRequest', tw.local.idjRequest); //BPD input variable.var name in bpd
tw.local.instanceID = tw.system.startProcessByName('CustomerProcess',tw.local.inputValues,true).id;
    var id = [1];
    if (tw.local.instanceID != null && tw.local.instanceID != '') {
        id= tw.local.instanceID.split(".");
    }
    tw.local.response = id[1];
 log.info("Exit Service by triggering BPM Instance ID:- " + " " + tw.local.instanceID);

To call the service
https://10.0.10.208:9444/rest/bpm/wle/v1/service/servicename?action=start&params=”+requestdata

To get the current instance ID

In your script:

tw.local.instanceid = tw.system.currentProcessInstance.id;
var id = [1];
if(tw.local.instanceid !=null &&tw.local.instanceid !=’’){
id=tw.local.instanceid.split(.);
}
tw.local.instanceid = id[1];

To terminate a bpm instance:

tw.system.findProcessInstanceByID(instanceid).abort();

API to get logged in user name:

tw.system.user.name

API to get all the teams user is in:

tw.system.user.teams

Script to check whether there is any active instance for a specific unique field in request using search column api. Here my unique id which Im searching is loanRef

log.info("----Validate Loan Reference existence for loan ref----" + tw.local.loanRef);
var col1 = new TWSearchColumn();
col1.name = TWSearchColumn.ProcessInstanceColumns.ID;
col1.type = TWSearchColumn.Types.ProcessInstance;

var col2 = new TWSearchColumn();
col2.name = TWSearchColumn.ProcessColumns.Name;
col2.type = TWSearchColumn.Types.Process;

var col3 = new TWSearchColumn();
col3.name = TWSearchColumn.ProcessInstanceColumns.Status;
col3.type = TWSearchColumn.Types.ProcessInstance;

var searchCondition = new TWSearchCondition();

var mySearchCol2 = new TWSearchColumn();
mySearchCol2.type = TWSearchColumn.Types.BusinessData;

mySearchCol2.name = "loanRef";
searchCondition.column = mySearchCol2;

searchCondition.operator = TWSearchCondition.Operations.Equals;
searchCondition.value = tw.local.loanRef;

// Search Ordering
var order1 = new TWSearchOrdering();
order1.column = col2;
order1.order = TWSearchOrdering.Orders.Descending;

// Create search object and set values
var search = new TWSearch();

search.columns = [col1, col2, col3];
search.organizedBy = TWSearch.OrganizeByTypes.ProcessInstance; //Task;
search.conditions = new Array(searchCondition);

// EXECUTE THE SEARCH IN THE CONTEXT OF USER change the limit of 20 as desired
var results = search.execute();

log.info('***print results length:- '+results.rows.length);

tw.local.results = new tw.object.listOf.BusinessReport();

if (results != null && results.rows != null && results.rows.length > 0) {
    log.info("Results not null and row length = " + results.rows.length);
    for (var i = 0; i < results.rows.length; i++) {
        var task = new tw.object.BusinessReport();
        task.id = results.rows[i].values[0];
        task.name = results.rows[i].values[1];
        task.status = results.rows[i].values[2];
        tw.local.results[i] = task;
     log.info("Instance Id: " + results.rows[i].values[0]);
     log.info("Process Name: " + results.rows[i].values[1]);
     log.info("Status: " + results.rows[i].values[2]);
    } //for
} //if
var len = tw.local.results.listLength;
if (tw.local.results != null && len > 0) {
    for (var i = 0; i < len; i++) {
        if (tw.local.results[i].status != "Completed") {
            tw.local.response = true;
            log.info("***Ref exists for PIID: " + tw.local.results[i].id);
        } else {
            tw.local.response = false;
            log.info("***Ref does NOT exists in BPM database***");
        }
    }
} else {
    tw.local.response = false;
    log.info("**Ref does NOT exists and executed in else loop ***");
}
log.info("Response Sent for IDJ: " + tw.local.response);

Doing rest calls from IBM BPM

In any real time project there would be a basic need to do some rest calls to external system apis , let it is be GET,POST,PUT etc. So it is important for Bpm developers to know how we can implement this.
To achieve this we can use available toolkit like Rest-Utilities , Kolban toolkit etc.

In this blog I will show you how we can achieve this using Kolban one. I won’t be importing the whole toolkit but I will just add the JAR file to server which is used for doing rest calls so that I reduce the size of my process application.

You can get the Jar file from my git project here

Once you download this Jar file to local, you need to upload this to bpm server

restjar

The idea here is to create a service using this JAR file , so that when ever there is a need to implement rest call, we can just use that service.

I have created a Integration service. Then since we need to call a Jar I have taken Java Integration from the palette to my diagram. Then in the definition section where we have Java class we need to select this JAR file and then we need to select the Method which we need to call from this JAR .In this Jar we have only doRest method so I have selected.

jarfile.png

In the image we can see that the method is expecting 6 parameters which we can see in the Data Mapping. Since these method is expecting these values I have created 6 input variables and then did the bindings and then as output I have created one variable as responseText
restservice

Now the service is ready and when ever there is a need to do rest call you can use this Execute Rest call service

I will show an example how we can do a POST call using the service which we created.First I need to create the variables which are necessary to be passed to the service.Here command is nothing but type of call .So it is “POST”. Next URL variable is created. Content variable is created to pass the data to that call in body and Header Map is the variable to pass headers.

testrest.png

Then we need to take a nested service and bind Execute Rest Call service . I need to pass headers to the service which I will pass in the scriptrestheaders

Then bind the input variables to the service. I don’t have userId and password , so I will ignore that. Also bind the output variable.

testservice.png

If you need to parse your response you can do that by adding script after the nested service and pass it as output variables like reponseCode ,responseMessage etc

In this blog I have used bpm 8.5.6 . Even in the higher versions we can implement this similarly with/without minor modifications.

This way we can easily implement rest calls from bpm

Please do comment if you have any questions . Hope this helps someone and have a good day!

 

Inserting and Retrieving records in to database(sql) using IBM BPM

In this blog I will show you how we can insert a record in to sql table and also retrieving the record from table.

Inserting data into database :

First we need to start with creating the service. Here we need to create Integration service.The values which we need to insert in to table will be obtained as an input to service.
input

Here I have only single complex object as input to service, but we can even have multiple input variables as input.Coming to the diagram , first the sql query needs to be written. For that I have used server scriptlet and I’ve binded that to sql variable. So that we can use that for bindings later.

sqlquery.png

Here I have used internal DB so there is no need of any schema for CRUD operations . But if we are connecting to external sql server we need to have db schema.

NOTE :  As a good practice DB schema needs to created as an epv variable and can be referenced in the service. So that any change in the schema details needs updation just at one place.

<# var schema = String(tw.epv.MHV_CONSTANT.INFLIGHT_DB_SCHEMA_NAME) + "." + String(tw.epv.MHV_CONSTANT.INFLIGHT_DB_GROUP); #>

INSERT INTO <#= schema #>.H_EmployeeDetails (*,*,*) VALUES (?,?,?)

Then we need  to initialize parameters as List of  SQL Parameter type as we need to pass that as input to SQL Execute Statement nested service.Then in this parameters we need to have initialization for all the values that we pass in our SQL query.

tw.local.parameters = new tw.object.listOf.SQLParameter();

tw.local.parameters[0] = new tw.object.SQLParameter();
tw.local.parameters[0].value = tw.local.employeeDetails.employeeId;
tw.local.parameters[0].mode = "IN";

tw.local.parameters[1] = new tw.object.SQLParameter();
tw.local.parameters[1].value = tw.local.employeeDetails.name;
tw.local.parameters[1].mode = "IN";

tw.local.parameters[2] = new tw.object.SQLParameter();
tw.local.parameters[2].value = tw.local.employeeDetails.place;
tw.local.parameters[2].mode = "IN";

setparams

Then we need to have the SQL Execute Statement Nested service. Then  we need to do the necessary bindings

binding

Now the service is ready for the insertion of data in to sql table.

Retrieving data from database :

Coming to retrieving data from table, It is similar to insertion but we need to mention the returnType , how we are expecting the result. It may be some business object type, XMLElement, NameValuePair etc., If it is other than business object type then we need to parse that response before sending that as output. I will show both the ways

  1. If return type is set as business Object type then results and output variable should also be of same business object type.retrieveinput.png

Initializing the params is same.We need to initialize the input variable here

tw.local.parameters = new tw.object.listOf.SQLParameter();

tw.local.parameters[0] = new tw.object.SQLParameter();
tw.local.parameters[0].value = tw.local.empId;
tw.local.parameters[0].mode = "IN";

bomapping

So we will have the response in results variable. Then you need to assign that to your output variable.It can done in the Post-execution Assignment.

tw.local.employeeData = tw.local.results[0]

2.If the return type is of type XMLElement , please check here

Please do comment if you have any questions . Hope this helps someone and have a good day!

Script to Parse XML in IBM BPM

We use XML parsing a lot during database calls. For retrieving data from a table it may return result of type “XMLElement”. In such cases we need to parse that in to an business object . Here I will show the same.

In my retrieve data integration service Im expecting result of type EmployeeDetails

Then in the Sql Excute nested service the return type is “XMLElement”

This above will return result of below type

This needs to be parsed as Business object to be sent as output

 

Parsing Script

if(tw.local.results != null && tw.local.results.childNodes && tw.local.results.childNodes.length > 0 ){
    var firstelement = tw.local.results.childNodes.item(0);
     tw.local.employeeData = new tw.object.EmployeeDetails();
     tw.local.employeeData.employeeId = firstelement.xpath("record/column[@name='EMPID']")[0].getText();
     tw.local.employeeData.name = firstelement.xpath("record/column[@name='EMPLOYEENAME']")[0].getText();
     tw.local.employeeData.place = firstelement.xpath("record/column[@name='PLACE']")[0].getText();
     tw.local.employeeData.phone = firstelement.xpath("record/column[@name='PHONE']")[0].getText();
     tw.local.employeeData.designation = firstelement.xpath("record/column[@name='DESIGNATION']")[0].getText();
     tw.local.employeeData.department = firstelement.xpath("record/column[@name='DEPARTMENT']")[0].getText();
     tw.local.employeeData.qualification = firstelement.xpath("record/column[@name='QUALIFICATION']")[0].getText();
     tw.local.employeeData.address = firstelement.xpath("record/column[@name='ADDRESS']")[0].getText();
}

This results in below output variable

 Same way we can parse for other return types such as NameValuePair etc.

Please do comment if you have any questions . Hope this helps someone and have a good day!

Auto Conversion of BPM TwObject to Json and Json to TwObject

Mostly in any development project we are using json for the data interchanging. Let it be with in the application or calling any external application. It is no difference in BPM .If there is any need to call external system it would be easy to use json rather than Tw objects,XML and vice versa for external systems to call BPM.

If you are using BPM 8.6 version there is an JSAPI  available for conversion of TWObject to JSON

Check Here

toJSONString( boolean formatted) Returns the JSON representation of the object

tw.local.json = tw.local.data.toJSONString(true);

But this api may not work for the lower BPM versions. Moreover there is no JSAPI available to convert JSON object to TWObject.

To solve this problem we can use BPM JSON helper toolkit to convert TWObjects to JSON and JSON to TwObjects and it is based on json2.js To keep our Process App size down,instead of entire helper kit we can just add 2 files to server as dependencies.

json2.js

BPM-JSON-Utils.js
Can be downloaded here : Git Download

server

Once we add these files now we can easily do what we required.

To convert TwObject to Json we need to call function convertTwToJSON(object) which is in Utils.js file

tw.local.json = BPMJSON.convertTwToJSON(tw.local.data);

Similarly to convert JSON to Twobject we use the below function

tw.local.data = BPMJSON.convertJSONToTw(tw.local.json);

Below is one sample example:

Initially I’m creating two integration services

service1

Here json variable is the output variable and data is the input variable to that service

service2

Here data is the output variable and json is the input variable to this service.
Then Ive created a simple human service so that I will call these services.

coach

Human service Diagram

 

coachoutput

Human service output

Here on click of convert to json the , JSONConvertor service is called and it returned the json. Same way we can use the json and convert to UI and directly bind that to UI.

NOTE : The point to be noted while converting json to Twobject , the output Tw object should exactly match with json structure or else it will throw an error.

This way we can easily work with Json even for complex business objects in bpm.

Hope it helps someone !