Bash Script Note : How to write your first Bash Script

How to write your first Bash Script

https://linuxconfig.org/bash-scripting-tutorial


1. Create  a file called hello.sh in promt:
   
        cat > hello.sh 

2. Edit content

     # Test file

3. press CTRL+D to save file

4. display contents, type

    cat hello.sh

5. Edit contents
 
   vi hello.sh

6. make the file executable

   chmod +x hello.sh

7. execute your first bash script

  ./hello.sh

Bash Script Note : redirection, pipe, variable

The Note is a quite review of what has been learnt on website :

http://tldp.org/HOWTO/Bash-Prog-Intro-HOWTO-3.html

The Note is driven by examples of short snippets. The brief command function is formatted as COMMAND:DESCRIPTION.

ECHO: the function is the same as printf


3. All about redirection

Stdout, Stderr to a file

This will cause the ouput of a program to be written to a file.

        ls -l > ls-l.txt

This will cause the stderr ouput of a program to be written to a file.

        grep da * 2> grep-errors.txt

4.Pipes 

Pipes let you use the output of a program as the input of another one.
This is very simple way to use pipes.

        ls -l | sed -e "s/[aeio]/u/g"   
        

Here, the following happens: first the command ls -l is executed, and it's output, instead of being printed, is sent (piped) to the sed program, which in turn, prints what it has to.
Probably, this is a more difficult way to do ls -l *.txt, but it is here for illustrating pipes, not for solving such listing dilema.

        ls -l | grep "\.txt$"
        

Here, the output of the program ls -l is sent to the grep program, which, in turn, will print lines which match the regex "\.txt$".

5.Variables
You can use variables as in any programming languages. There are no data types. A variable in bash can contain a number, a character, a string of characters.
You have no need to declare a variable, just assigning a value to its reference will create it.

            STR="Hello World!"
            echo $STR    
            

Line 2 creates a variable called STR and assigns the string "Hello World!" to it. Then the VALUE of this variable is retrieved by putting the '$' in at the beginning. Please notice (try it!) that if you don't use the '$' sign, the output of the program will be different, and probably not what you want it to be.


A very simple backup script

           #!/bin/bash          
           OF=/var/my-backup-$(date +%Y%m%d).tgz
           tar -cZf $OF /home/me/
           

This script introduces another thing. First of all, you should be familiarized with the variable creation and assignation on line 2. Notice the expression '$(date +%Y%m%d)'. If you run the script you'll notice that it runs the command inside the parenthesis, capturing its output.

Notice that in this script, the output filename will be different every day, due to the format switch to the date command(+%Y%m%d). You can change this by specifying a different format.
Some more examples:
echo ls
echo $(ls)

Local variables can be created by using the keyword local.

                #!/bin/bash
                HELLO=Hello 
                function hello {
                        local HELLO=World
                        echo $HELLO
                }
                echo $HELLO
                hello
                echo $HELLO

Nokogiri error

How to use Nokogiri to collect data?

The first step is to Install Nokogiri 

When I tried to install Nokogiri on my Windows OS using


gem install nokogiri


I confronted an error shows

 It looks like it is about the conversion of the binary code. I can imagine a real computer engineer could solve this with casual enthusiasm, but now I have to count on myself. 

 I turn to Google of help. Nothing.

Then I turn to Nokogiri-talk , no helpful information. 

 I post my question on it , and I am suggested to upgrade my RDoc packet. The suggestion really helped!

 And I found on the blog :

 Installing libxml-ruby on Windows: The painless way 

 that it is prerequisite to install libxml or libxml before installing Nokogiri. Do what it ask you to do.

  

but I encounter another trouble, 

when I typed    irb(main):001:0> require 'rubygems'

an error occured...

Then, we can get started with Nogokiri !

We can get example of Nokogiri by googling "Nokogiri css example"

Ruby Programming Language: Basic Ruby command

We have learned how to run Ruby on cmd.exe on Windows

So let's learn some Ruby.

Concise Ruby Language Tutorial 
A short, but quite enough tutorial for beginners. But we don't spend too much time going through every chapter here, because our goal is to learn data-mining, not Ruby. We can review it when we need it.


Quick Note
 
A quick overview of the most basic commands, in case we forgot.

   

Ruby Programming Language Note: Basic Commands for cmd.exe

Ruby Programming Language Note

The first class of learning ruby is not learning ruby language, but leaning some command for cmd.exe. It's because we data miners don't use Interactive Ruby, but Command Prompt Ruby.

we run ruby on cmd.exe.

This is the first class I learned 

command                           description
---------------------------------------------------------------------------------------------------
dir                                      list the files/directory under the current directory
                                          Its equivalent is ls on Linux and OS

cd                                      enter the directory/ file

cd ..                                   return to the previous directory, remember the space before tow
                                         periods.  // very similar to Linux commnad ^^
mkdir                                 make directory

move                                 move

copy                                 copy

ruby                                 run ruby files

You can find all commands and its explanation HERE

Tutorial for cmd.exe commands

For people who haven't learn Linux before, I recommend the Bible of Linux- 鳥哥的Linux私房菜

------------------------------------------------------------------------------------------------------

OK, this is the first class here.

Data mining booknote

Data mining takes advantage of advances in the fields of artificial intelligence (AI) and statistics.Both disciplines have been working on problems of pattern recognition and classification. Both communities have made great contributions to the understanding and application of neural nets and decision trees.

1. Describe the Data
        explore data, select data, cleanse the data,and classify data.
 
2. Build Predictive Model
          I. use sample data to build a predictive model (based on patterns with known results)
         II. use data outside the sample to test the veracity of the model
        III. empirically verify the model by applying model to customer's database

3.  Data Mining's DONTS
        I. It doesn't uncover solution automatically, and the patterns uncovered must be verified to
            the reality.
       II. the predictive uncovered patterns are NOT necessarily the CAUSES of the behaviors.
      III. you must understand your data,

4. Data mining does not replace skilled business analysts but confirm their empirical observations
   and find new, subtle patterns that yield steady incremental improvement (plus the occasional
   breakthrough insight).


Application:
    Detect Fraud:  Telecommunication, Credit Card Company, Insurance Company, and Stock
                          Exchange Company.


   Medical effective test


   Retailer





     









source: http://www.twocrows.com/intro-dm.pdf

Software for Data Mining

To Start Data Mining, You Need:

MicroSoft Excel  

  the most available software, and just avoid us from wasting time on learning the interface.

mySQL

R
  a statistic analysis software. Free and professional. Everyone deals with statistics use it            
  Download point http:// cran.r-project.org/


RUBY

  language that makes it easy to grab data table and transform data into suitable format you need.
  easy to learn and use.

Nokogiri
 
   Nokogiri (鋸) is an HTML, XML, SAX, and Reader parser. Among Nokogiri’s many features is the ability to search documents via XPath or CSS3 selectors.
 
Nokogiri save files in a Ruby library, and that's why we use ruby.