Lucy API is a Python API for building smart conversation agent. It allows companies to handle messages sent to their chat channels automatically. Â
This tutorial is to help you install and configure necessary packages for Python for the data mining class. Contact me (inssong@gmail.com) if you have any questions following the steps to prepare for your assignments.
Overview of the installation of Lucy API:
Video tutorial. (If the video does not playback properly, please try out Google Chrome.).
Installation of Python and Data Mining Packages
First, download and install Python 2.7Â (install 32-bit version) on your computer.
Video Tutorial
Next, download and install PyCharm IDE from JetBrains. It is free for students. Make sure to sign up for the free student license.
Video tutorial
Now, let's install some basic packages for data mining.
Numpy package is often required for various data mining modules. However, installation is a bit tricky because you need to install suitable binary files for your OS.
- For Windows, go to this site and download the latest binary package for Python 2.7, something called "..superpack...python 2.7..."
- Download exe file. Video tutorial for this step.
- Select Python 2.7 that you installed when asked.
- For Mac, skip this. You already have Numpy installed for python 2.7.
SciPy package provides useful tools for scientific computing and plotting. Installation is similar to Numpy.
- For Windows, go to this site and download the latest binary package for Python 2.7, something called "..superpack..python 2.7..."
- For Windows, download exe file.
- Run the downloaded installation file.
- Video tutorial for this step.
- For Mac, skip this. You already have Numpy installed for python 2.7.
Matplotlib is a convenient tool if you need to plot anything. This is optional, so install it if you need.
- Open the command line tool (DOS or Terminal)
- If you installed Python in C:/python27, type
- c:/python27/python -m pip install matplotlib
- Video tutorial.
scikit-learn is a Python machine learning package with excellent documentation in learning data mining algorithms.
- On Windows, once you have installed Numpy and SciPy, install it by typing the following in the command line:
- c:/python27/python -m pip install -U scikit-learn
- Video tutorial for this step.
- On Mac, numpy and scipy can be installed together (not tested)
- python pip install -U numpy scipy scikit-learn
- If you see an error saying that pip is not installed, install pip
- Additionally install Scikit-image for image processing
NLTK is a natural language processing package for Python
- After installing Numpy, type the following command line:
- c:/python27/python -m pip install nltk
- Video tutorial for this step.
Check Installation
Now, we have the necessary tools installed, let's test if the tools are correctly installed. Open PyCharm, create a new Python project and add a python file.
# Numpy test
import numpy as np
parray = [1.2, 2, 2]  # this is python array
C = np.array(parray) Â # a matrix, a vector
print C*2 + 10  # Matlab like matrix operation
Run the Python script in Pycharm (type Shift+Ctrl+F10 in the editor).
Try a classification example from scikit-learn.
Build a Message Handling Robot for Your Chapter
KOPO MES allows users to create their own journeys and chapters for the journeys. A chapter is a personalized chat channel for students. We can use Python to handle messages sent by students for the chapters.
- First, create an account at KOPO MES if you haven't done already.
- With the account, we can then create a journey and a chapter for the journey. To handle the messages manually or using Python, we do not add any questions for the chapter.
- Select Lucy Agent mode for the chapter type to handle chat messages using your agent script.
Now, we need to get the ID of the chapter. We can get this from the chapter list page of the journey. Click on the column icon on the top-left of the table and check ID to show IDs of the chapters. We need this ID to identify chapters in the Python program. Video tutorial.
Now, create another account for your Lucy Agent Bot and share the chapter with the Lucy Agent bot using the email address of the bot. ** Make sure to give Edit permission.
- Click on share setting for specific people:
- Add your new Lucy Agent bot account to the share list:
- Now use the Lucy Agent bot account to run the agent program. Use the bot account details to set the username and password.
Follow the following steps to download, configure and run the Lucy Bot agent on your computer.
- Download Lucy Agent API (Version 2018-07-09). (Old version: LucyAgent.zip). Â
- Add the Python files to your project folder and open lucy_echo.py.
- Install necessary packages using python pip:
- python -m pip install poster
- You can also open message.py in the library and resolve the missing libraries from PyCharm:
Right click on each missing libraries and install
- Change the username, password, lesson_id (chapter ID). Make sure to use the Lucy agent bot account for these details.
- Run the code (shift-ctrl-F10).
- Ask friends to enroll in the journey and the chapter, and send some messages.
** Note: the robot cannot reply to the same account, otherwise it will keep replying to itself.
- See how your program reply to the messages sent by your friends. Â
Try a demo of Lucy Bot on "Lucy Bot Echo Demo" of this journey. If the bot is running, it will echo back your messages.
We can now run our very own conversation agent. Just add more intelligent message handlers using machine learning algorithms. Note: the account for Lucy Agent must have an edit right for the chapter to read and reply to the messages for the chapter.
Video tutorial on running Lucy conversation agent.
Sentence Classifier for the Robot Agent
Let's build a classifier that can predict if a sentence is an open question or a closed question.
Although this sounds a trivial task, you will soon find out that that not the case. The number of possible sentences in the world can be too many. We ourselves can classify sentences fairly easily because we have been trained on it for over 10 or 20 years. However, we cannot wait 10 years for our program to learn to classify sentences.
See lucy_predict.py that uses an SVM classifier to predict whether a sentence is an open question or not.
It uses the provided vocabulary to convert a sentence to a set of features, which are then used to predict if the sentence is an open question.
Clustering Sentence
What if we do not know what kind of messages are sent from users. Certainly, there are many more types of sentences other than questions and statements. Instead of listing all sentence types, we can cluster messages and use the clusters as class labels and handle messages accordingly.
Build your database of messages and cluster the messages using a clustering algorithm.
Testing LucyAPI on Web Applications
To call LucyAPI services from a web application (e.g., HTML5 application), you will need to fix "Access-Control-Allow-Origin' error. On Google Chrome, you can install a plugin to solve this problem.
Future of Robot Conversation Agent
Now, think about what else can you do to make the conversation agent helpful for students?
Lucy Agents developed using Lucy Agent API:
Additional Packages
Pillow (Python Imaging Library. This is a replacement of PIL. Do not install PIL): python -m pip install pillow
Word Cloud: python -m pip install wordcloud
The following trainers are available for hire for your organizations:
- Albert Hardanto: Expert Python programmer and certified Lucy Agent API developer.
- Thanh Tam Nguyen: Expert Python programmer and certified Lucy Agent API developer.
- Priyanka Rana: Expert Python programmer and certified Lucy Agent API developer.
Please contact Dr. Insu Song (inssong@gmail.com) if you would like to arrange training sessions with our experts.
References
Other useful links for your data mining carrier.
LAPACK (Linear Algebra Package): http://netlib.org/lapack/lug/lapack_lug.html
HPC Challenge Benchmarks:Â http://icl.cs.utk.edu/hpcc/
Top 500 Super Computers:Â http://www.top500.org/
Using Python in Hadoop: http://www.michael-noll.com/tutorials/writing-an-hadoop-mapreduce-program-in-python/
NLTK for Python (Natural Language Processing): http://www.nltk.org/book/
Using R on Hadoop: http://hortonworks.com/hadoop-tutorial/using-rhadoop-to-predict-visitors-amount/Â
