Thursday, July 22, 2021

How to transfer the data from one db to another using azure datafactory ( ADF ) ?

 Steps:

1) Create Azure Datafactory 

2)  Create Linked service 

3) Add the dataset to the linked service.


az cosmosdb list-connection-strings --name <> --resource-group ..
az datafactory linked-service create --resource-group <> --factory-name <> --properties "@MongoDBAPILinkedService.json" --name TigerLinkedSource
@MongoDBAPILinkedService.json
{
"type": "CosmosDbMongoDbApi",
"typeProperties": {
"connectionString": "mongodb://**********:primarypassword@*****************.documents.azure.com:10255/?ssl=true&replicaSet=globaldb",
"database": "tiger-db"
}
}
step3:
adding the dataset
// source dataset az datafactory dataset create --resource-group <> --factory-name  <> --dataset-name OutputDataset --properties "@MongoDBAPIDatasetSource.json"{
"linkedServiceName": {
"referenceName": "TigerLinkedSource",
"type": "LinkedServiceReference"
},
"annotations": [],
"type": "CosmosDbMongoDbApiCollection",
"schema": [],
"typeProperties": {
"collection": "tiger"
}
}
Please do the above process for the sink.
Create a pipeline in between the source and sink & trigger/run the pipeline using azure cli.
az datafactory pipeline create --resource-group <> --factory-name <> --name Pipeline --pipeline "@Pipeline.json"
Pipeline.json
{
"activities": [
{
"name": "CopyFromSourceToSink",
"type": "Copy",
"dependsOn": [],
"policy": {
"timeout": "7.00:00:00",
"retry": 0,
"retryIntervalInSeconds": 30,
"secureOutput": false,
"secureInput": false
},
"userProperties": [],
"typeProperties": {
"source": {
"type": "CosmosDbMongoDbApiSource",
"batchSize":100
},
"sink": {
"type": "CosmosDbMongoDbApiSink",
"writeBatchTimeout": "00:30:00",
"writeBehavior": "insert"
},
"enableStaging": false
},
"inputs": [
{
"referenceName": "InputDataset",
"type": "DatasetReference"
}
],
"outputs": [
{
"referenceName": "OutputDataset",
"type": "DatasetReference"
}
]
}
]
}
trigger/run the pipeline using this azure cli command.
az datafactory pipeline create-run --resource-group <> --factory-name <> --name Pipeline



Saturday, June 19, 2021

how to measure the trained model ?

 Azure ML uses model evaluation for the measurement of the trained model accurancy.

For classification models,  the evaluate model module provides the following five metrics:

1. Accurancy 

2. precision

3. Recall

4. F1 score

5. Area under curve ( AUC ).



AZURE ML DESGINER

 two data sources for the import data modules in the Azure ML Designer.

datastore and URL via http.

what is recall metric ?

 Recall metrics define how many positive cases that model predicted are actually predicted right . We can calculate this metric using the following formula 

        TP/( TP + FN )

confusion matrix of the model

 what is the expression for model precision value calculation ?




formula :    TP/(TP + FP )

EXPRESSION :  577 / ( 577 + 245 ).






Azure ML studio home screen ?

 Three main authoring tools:

 1. Notebooks.

 2. Designer.

 3. Automated ML 


what Microsoft Bot Framework supports ?

 supports two models of bot integration with agent engagement platforms like customer support service.

These two models are Bot as agent and Bot as proxy.

Bot as agent model integrates bot on the same level as live agents. The bot is engaged in interactions the same way as customer support personnel. 

Handoff  protocol regulates bot's disengagement and a transfer of user's communication to a live person.

This is the most straight forward model to implement.

Bot as proxy model integrates bot as the primary filter before the user interacts with a live's agent.

Bot's logic decides when to transfer a conversation and where to route it. This model is more complicated to implement.