2018/2/12

Apache HTTP Client 4.5 Fundamentals


Apache HTTP Client 是目前最常被使用的 Java HTTP Client Library,他經歷了許多次改版,目前的正式版為 4.5,5.0 還在測試階段。HttpClient 沒有完整的瀏覽器的功能,最大的差異是缺少了 UI,他單純地只有提供 HTTP Protocol 1.0 及 1.1 的資料傳輸及互動的功能,通常用在 Server Side,需要對其他有 HTTP 介面的 Server 進行資料傳輸互動時,例如在 Server 對 Google 發送搜尋的 reqest,並對 Google 回應的 HTML 內容進行解析。


除了 HTTP Request 及 Response 的 Messages 以外,所有 HTTP Request 都要以某一個 HTTP Method 的形式發送給 Server,最常用的是 GET 及 POST,在 HTTP Message 中會有多個 Headers 描述 metadatas,而在 Response 中,會夾帶可儲存在 Client 的 Cookie 資料,並在下一次的 Request 回傳給 Server,Session 是指一連串的多個 Http Request 及 Response 的互動過程,通常會以 Cookie 的方式紀錄 Session ID。


HTTP Fundamentals


這是最基本的 HttpGet


CloseableHttpClient httpclient = HttpClients.createDefault();
HttpGet httpget = new HttpGet("http://localhost/");
CloseableHttpResponse response = null;

try {
    response = httpclient.execute(httpget);
} catch (IOException e) {
    e.printStackTrace();

    logger.error("error: ", e);
} finally {
    try {
        response.close();
    } catch (IOException e) {
        e.printStackTrace();

        logger.error("error: ", e);
    }
}



Http methods 有 GET, HEAD, POST, PUT, DELETE, TRACE and OPTIONS,針對每一種 method 都有提供專屬的 class: HttpGet,
HttpHead, HttpPost, HttpPut, HttpDelete, HttpTrace, and HttpOptions。


Request URI 是 Uniform Resource Identifier,可識別資源的位置,HTTP Request URI 包含了 protocol scheme, host name, optional port, resource path,
optional query, and optional fragment 這幾個部分,可用 URIBuilder 產生 URI。


// uri=http://www.google.com/search?q=httpclient&btnG=Google+Search&aq=f&oq=
URI uri = new URIBuilder()
                    .setScheme("http")
                    .setHost("www.google.com")
                    .setPath("/search")
                    .setParameter("q", "httpclient")
                    .setParameter("btnG", "Google Search")
                    .setParameter("aq", "f")
                    .setParameter("oq", "")
                    .build();
HttpGet httpget = new HttpGet(uri);



HTTP response


HTTP response 是 server 回傳給 client 的 message。


HttpResponse httpResponse = new BasicHttpResponse(HttpVersion.HTTP_1_1,
        HttpStatus.SC_OK, "OK");
System.out.println(httpResponse.getProtocolVersion());
System.out.println(httpResponse.getStatusLine().getStatusCode());
System.out.println(httpResponse.getStatusLine().getReasonPhrase());
System.out.println(httpResponse.getStatusLine().toString());
            


HttpResponse httpResponse2 = new BasicHttpResponse(HttpVersion.HTTP_1_1,
        HttpStatus.SC_OK, "OK");
httpResponse2.addHeader("Set-Cookie",
        "c1=a; path=/; domain=localhost");
httpResponse2.addHeader("Set-Cookie",
        "c2=b; path=\"/\", c3=c; domain=\"localhost\"");
Header h1 = httpResponse2.getFirstHeader("Set-Cookie");
System.out.println(h1);
Header h2 = httpResponse2.getLastHeader("Set-Cookie");
System.out.println(h2);
Header[] hs = httpResponse2.getHeaders("Set-Cookie");
System.out.println(hs.length);

輸出結果


HTTP/1.1
200
OK
HTTP/1.1 200 OK


Set-Cookie: c1=a; path=/; domain=localhost
Set-Cookie: c2=b; path="/", c3=c; domain="localhost"
2

http message 中包含了許多 headers,可利用 HeaderIterator 逐項處理每一個 header,另外有一個 BasicHeaderElementIterator 可以針對某一種 header,處理所有 header elements。


HttpResponse response = new BasicHttpResponse(HttpVersion.HTTP_1_1,
        HttpStatus.SC_OK, "OK");
response.addHeader("Set-Cookie",
        "c1=a; path=/; domain=localhost");
response.addHeader("Set-Cookie",
        "c2=b; path=\"/\", c3=c; domain=\"localhost\"");

// HeaderIterator
HeaderIterator it = response.headerIterator("Set-Cookie");
while (it.hasNext()) {
    System.out.println(it.next());
}

// HeaderElementIterator
HeaderElementIterator it2 = new BasicHeaderElementIterator(
        response.headerIterator("Set-Cookie"));
while (it2.hasNext()) {
    HeaderElement elem = it2.nextElement();
    System.out.println(elem.getName() + " = " + elem.getValue());
    NameValuePair[] params = elem.getParameters();
    for (int i = 0; i < params.length; i++) {
        System.out.println(" " + params[i]);
    }
}



HTTP entity


HTTP message 能封裝某個 request/response 的某些 content,可在某些 request/response 中找到,他是 optional 的資料。使用 entities 的 request 稱為 entity enclosing requests,HTTP request 中有兩種 entity request methods: POST 及 PUT。


除了回應 HEAD method 的 response 以及 204 No Content, 304 Not Modified, 205 Reset Content 以外,大部分的 response 較常使用 entity。


HttpClient 區分了三種 entities: streamed, self-contained, wrapping,通常會將 non-repeatable entities 視為 streamed,而將 repeatable entities 視為 self-contained。


  1. streamed: content 是由 stream 取得,常用在 response,streamed entities 不能重複。

  2. self-contained: content 存放在記憶體中,或是由 connection 以外的方式取得的,這種 entity 可以重複,通常用在 entity enclosing HTTP requests。repeatable 就是可以重複讀取 content 的 entity,ex: ByteArrayEntity or StringEntity。

  3. wrapping: 由另一個 entity 取得的 content


因 entity 可存放 binary 及 character content,因此支援了 character encodings。


可利用 HttpEntity#getContentType(), HttpEntity#getContentLength() 取得 Content-Type and Content-Length 欄位的資訊,因 Content-Type 包含了 character encoding 的資訊,可用 HttpEntity#getContentEncoding() 取得,如果 HttpEntity 包含了 Content-Type header,就能取得 Header 物件。


StringEntity myEntity = new StringEntity("important message",
        ContentType.create("text/plain", "UTF-8"));
    
System.out.println(myEntity.getContentType());
System.out.println(myEntity.getContentLength());
System.out.println(EntityUtils.toString(myEntity));
System.out.println(EntityUtils.toByteArray(myEntity).length);

結果


Content-Type: text/plain; charset=UTF-8
17
important message
17



Ensuring release of low level resources


為確保系統資源有回收,必須要關閉 entity 取得的 content stream,或是直接關閉 response,關閉 stream 時,還能保持 connection,但如果關閉 response,就直接關閉並 discards connection。


CloseableHttpClient httpclient = HttpClients.createDefault();
HttpGet httpget = new HttpGet("http://localhost/");
CloseableHttpResponse response = httpclient.execute(httpget);
try {
    HttpEntity entity = response.getEntity();
    if (entity != null) {
        InputStream instream = entity.getContent();
        try {
            // do something useful
        } finally {
            instream.close();
        }
    }
} finally {
    response.close();
}

HttpEntity#writeTo(OutputStream) 也能用來保證在 entity 完全寫入後, resource 能被釋放。如果是用 HttpEntity#getContent() 取得了 java.io.InputStream,就必須自行在 finally 中 close stream。如果是處理 streaming entities,使用 EntityUtils#consume(HttpEntity) 可保證 entity content 能完全被處理並回收 stream。

如果只需要處理部分 response content,可直接呼叫 response.close,就不需要消化所有的 response content,但 connection 也無法被 reused。


CloseableHttpClient httpclient = HttpClients.createDefault();
HttpGet httpget = new HttpGet("http://localhost/");
CloseableHttpResponse response = httpclient.execute(httpget);
try {
    HttpEntity entity = response.getEntity();
    if (entity != null) {
        InputStream instream = entity.getContent();
        int byteOne = instream.read();
        int byteTwo = instream.read();
        
        // Do not need the rest
    }
} finally {
    response.close();
}



Consuming entity content


最好的方式是呼叫 HttpEntity#getContent() 或是 HttpEntity#wrtieTo(OutputStream),但 HttpClient 同時也提供 EntityUtils 類別有多個處理 content 的 static methods,不建議使用 EntityUtils,除非 response 是由 trusted HTTP server 回傳,且是有限的長度。


CloseableHttpClient httpclient = HttpClients.createDefault();
HttpGet httpget = new HttpGet("http://localhost/");
CloseableHttpResponse response = httpclient.execute(httpget);
try {
    HttpEntity entity = response.getEntity();
    if (entity != null) {
        long len = entity.getContentLength();
        if (len != -1 && len < 2048) {
            System.out.println(EntityUtils.toString(entity));
        } else {
            // Stream content out
        }
    }
} finally {
    response.close();
}

如果需要多次讀取整個 entity content,最簡單的方式是以 BufferedHttpEntity class 封裝原本的 entity,這可讓 content 放如 in-memory buffer。


CloseableHttpResponse response = <...>
HttpEntity entity = response.getEntity();
if (entity != null) {
    entity = new BufferedHttpEntity(entity);
}



Producing entity content


StringEntity, ByteArrayEntity, InputSreamEntity, FileEntity 可用來透過 HTTP connetion stream out 資料。InputStreamEntity 只能被使用一次,不能重複讀取資料。


File file = new File("somefile.txt");
FileEntity entity = new FileEntity(file,
        ContentType.create("text/plain", "UTF-8"));
HttpPost httppost = new HttpPost("http://localhost/action.do");
httppost.setEntity(entity);



HTML forms


UrlEncodedFormEntity 模擬 submitting an HTML form。以下等同用 POST method 發送 param1=value1&param2=value2。


List<NameValuePair> formparams = new ArrayList<NameValuePair>();
formparams.add(new BasicNameValuePair("param1", "value1"));
formparams.add(new BasicNameValuePair("param2", "value2"));
UrlEncodedFormEntity entity = new UrlEncodedFormEntity(formparams, Consts.UTF_8);

HttpPost httppost = new HttpPost("http://localhost/handler.do");
httppost.setEntity(entity);



Content chunking


可直接呼叫 HttpEntity#setChunked(true) 建議分塊處理 content,但如果遇到不支援的 HTTP/1.0,還是會忽略這個設定值。


StringEntity entity = new StringEntity("important message",
ContentType.create("plain/text", Consts.UTF_8));
entity.setChunked(true);
HttpPost httppost = new HttpPost("http://localhost/acrtion.do");
httppost.setEntity(entity);



response handlers


透過 ResponseHandler interface 的 handleResponse(HttpResponse response) 這個方法處理 response 這個方式最簡單,programmer 不需要處理 connection management,HttpClient 會自動確保 connection 回到 connection manager。


public static void main(String[] args) {
    try {
        CloseableHttpClient httpClient = HttpClients.createDefault();
        HttpGet httpget = new HttpGet("http://localhost/json");

        ResponseHandler<MyJsonObject> rh = new ResponseHandler<MyJsonObject>() {
            public MyJsonObject handleResponse(final HttpResponse response) throws IOException {
                StatusLine statusLine = response.getStatusLine();
                HttpEntity entity = response.getEntity();
                if (statusLine.getStatusCode() >= 300) {
                    throw new HttpResponseException(statusLine.getStatusCode(), statusLine.getReasonPhrase());
                }
                if (entity == null) {
                    throw new ClientProtocolException("Response contains no content");
                }
                Gson gson = new GsonBuilder().create();
                Reader reader = new InputStreamReader(entity.getContent(), ContentType.getOrDefault(entity)
                        .getCharset());
                return gson.fromJson(reader, MyJsonObject.class);
            }
        };
        MyJsonObject myjson = httpClient.execute(httpget, rh);
        System.out.println(myjson.toString());

    } catch (Exception e) {
        e.printStackTrace();
    }
}

public class MyJsonObject {

}

HttpClient interface


HttpClient 是 thread safe,包含了多個 handler 及 strategy interface implementations,可以自訂 HttpClient。


ConnectionKeepAliveStrategy keepAliveStrat = new DefaultConnectionKeepAliveStrategy() {
    @Override
    public long getKeepAliveDuration(
            HttpResponse response,
            HttpContext context) {
        long keepAlive = super.getKeepAliveDuration(response, context);
        if (keepAlive == -1) {
            // Keep connections alive 5 seconds if a keep-alive value
            // has not be explicitly set by the server
            keepAlive = 5000;
        }
        return keepAlive;
    }
};
CloseableHttpClient httpclient = HttpClients.custom()
        .setKeepAliveStrategy(keepAliveStrat)
        .build();

如果 CloseableHttpClient 已經不需要使用了,且不要再被 connection manager 管理,就必須要呼叫 CloseableHttpClient#close()


CloseableHttpClient httpclient = HttpClients.createDefault();
try {
    <...>
} finally {
    httpclient.close();
}

HTTP execution context


HTTP 是 stateless, response-request protocol,但實際上 application 需要在數個 request-response 之間保存 state information。HTTP context functions 類似 java.util.Map 的概念,

HttpClient 4.x 可以維持 HTTP session,只要使用同一個 HttpClient 且未關閉連接,則可以使用相同會話來訪問其他要求登錄驗證的服務。


如果需要使用 HttpClient Pool,並且想要做到一次登錄的會話供多個HttpClient連接使用,就需要自己保存 session information。因為客戶端的會話信息是保存在cookie中的(JSESSIONID),所以只需要將登錄成功返回的 cookie 複製到各個HttpClient 使用即可。


使用 Cookie 的方法有3種,可使用同一個 HttpClient,可以自己使用CookieStore來保存,也可以通過HttpClientContext上下文來維持。


  • 使用同一個 CloseableHttpClient

public class TestHttpClient {

    public static void main(String[] args) {
        TestHttpClient test = new TestHttpClient();

        try {
            test.testTheSameHttpClient();

        } catch (Exception e) {
            e.printStackTrace();
        }

    }

    String loginUrl = "http://192.168.1.24/admin/config.php";
    String testUrl = "http://192.168.1.24/admin/ajax.php?module=core&command=getExtensionGrid";

    public void testTheSameHttpClient() throws Exception {
        System.out.println("----testTheSameHttpClient");

        //// 由 HttpClientBuilder 產生 CloseableHttpClient
        // HttpClientBuilder httpClientBuilder = HttpClientBuilder.create();
        // CloseableHttpClient client = httpClientBuilder.build();

        //// 直接產生 CloseableHttpClient
        CloseableHttpClient client = HttpClients.createDefault();

        HttpPost httpPost = new HttpPost(loginUrl);
        Map parameterMap = new HashMap();
        parameterMap.put("username", "admin");
        parameterMap.put("password", "password");

        UrlEncodedFormEntity postEntity = new UrlEncodedFormEntity(
                getParam(parameterMap), "UTF-8");
        httpPost.setEntity(postEntity);

        System.out.println("request line:" + httpPost.getRequestLine());
        try {
            // 執行post請求
            CloseableHttpResponse httpResponse = client.execute(httpPost);

            boolean loginFailedFlag = false;
            try {
                String responseString = printResponse(httpResponse);

                loginFailedFlag = responseString.contains("Please correct the following errors");

            } finally {
                httpResponse.close();
            }
            System.out.println("loginFailedFlag?:" + loginFailedFlag);

            if( !loginFailedFlag ) {
                // 執行get請求
                System.out.println("----the same client");
                HttpGet httpGet = new HttpGet(testUrl);
                System.out.println("request line:" + httpGet.getRequestLine());
                CloseableHttpResponse httpResponse1 = client.execute(httpGet);

                try {
                    printResponse(httpResponse1);
                } finally {
                    httpResponse1.close();
                }
            }

        } catch (IOException e) {
            e.printStackTrace();
        } finally {
            try {
                // close HttpClient and release all system resources
                client.close();
            } catch (IOException e) {
                e.printStackTrace();
            }
        }
    }

    private static String printResponse(HttpResponse httpResponse)
            throws ParseException, IOException {
        HttpEntity entity = httpResponse.getEntity();
        // response status code
        System.out.println("status:" + httpResponse.getStatusLine());
        System.out.println("headers:");
        HeaderIterator iterator = httpResponse.headerIterator();
        while (iterator.hasNext()) {
            System.out.println("\t" + iterator.next());
        }
        // 判斷 response entity 是否 null
        String responseString = null;
        if (entity != null) {
            responseString = EntityUtils.toString(entity);
            System.out.println("response length:" + responseString.length());
            System.out.println("response content:"
                    + responseString.replace("\r\n", ""));
        }

        return responseString;
    }

    private static List<NameValuePair> getParam(Map parameterMap) {
        List<NameValuePair> param = new ArrayList<NameValuePair>();
        Iterator it = parameterMap.entrySet().iterator();
        while (it.hasNext()) {
            Entry parmEntry = (Entry) it.next();
            param.add(new BasicNameValuePair((String) parmEntry.getKey(),
                    (String) parmEntry.getValue()));
        }
        return param;
    }
}

  • 使用 HttpContext

HttpContext 能夠保存任意的物件,因此在兩個不同的 thread 中共享上下文是不安全的,建議每個線程都一個它自己執行的context。


在執行 HTTP request 時,HttpClient 會將以下屬性放到 context 中


  1. HttpConnection instance: 代表連接到目標服務器的當前 connection。
  2. HttpHost instance: 代表當前 connection連接到的目標 server
  3. HttpRoute instance: 完整的連線路由
  4. HttpRequest instance: 代表了當前的HTTP request。HttpRequest object 在 context 中總是準確代表了狀態信息,因為它已經發送給了服務器。 預設的HTTP/1.0 和 HTTP/1.1使用相對的請求URI,但以non-tunneling模式通過代理發送 request 時,URI會是絕對的。
  5. HttpResponse instance: 代表當前的 HTTP response。
  6. java.lang.Boolean object 是一個標識,它標誌著當前請求是否完整地傳輸給連接目標。
  7. RequestConfig object: 代表當前請求配置
  8. java.util.List object: 代表一個含有執行請求過程中所有的重定向地址。

public class TestHttpContext {

    public static void main(String[] args) {
        TestHttpContext test = new TestHttpContext();

        try {
            test.testHttpContext();

        } catch (Exception e) {
            e.printStackTrace();
        }

    }

    String loginUrl = "http://192.168.1.24/admin/config.php";
    String testUrl = "http://192.168.1.24/admin/ajax.php?module=core&command=getExtensionGrid&type=all&order=asc";

    public void testHttpContext() throws Exception {
        System.out.println("----testHttpContext");

        //// 由 HttpClientBuilder 產生 CloseableHttpClient
        // HttpClientBuilder httpClientBuilder = HttpClientBuilder.create();
        // CloseableHttpClient client = httpClientBuilder.build();

        //// 直接產生 CloseableHttpClient
        CloseableHttpClient client = HttpClients.createDefault();

        // Create a local instance of cookie store
        CookieStore cookieStore = new BasicCookieStore();

        // Create local HTTP context
        HttpClientContext localContext = HttpClientContext.create();
        localContext.setCookieStore(cookieStore);


        HttpPost httpPost = new HttpPost(loginUrl);
        Map parameterMap = new HashMap();
        parameterMap.put("username", "admin");
        parameterMap.put("password", "max168kit");

        UrlEncodedFormEntity postEntity = new UrlEncodedFormEntity(
                getParam(parameterMap), "UTF-8");
        httpPost.setEntity(postEntity);

        System.out.println("request line:" + httpPost.getRequestLine());
        try {

            CloseableHttpResponse httpResponse = client.execute(httpPost, localContext);

            boolean loginFailedFlag = false;
            try {
                String responseString = printResponse(httpResponse, cookieStore);

                loginFailedFlag = responseString.contains("Please correct the following errors");

            } finally {
                httpResponse.close();
            }

            System.out.println("loginFailedFlag?:" + loginFailedFlag);

            if( !loginFailedFlag ) {
                // 使用新的 CloseableHttpClient
                CloseableHttpClient client2 = HttpClients.createDefault();

                // 執行get請求
                HttpGet httpGet = new HttpGet(testUrl);
                System.out.println("request line:" + httpGet.getRequestLine());
                CloseableHttpResponse httpResponse2 = client2.execute(httpGet, localContext);

                try {
                    printResponse(httpResponse2, cookieStore);
                } finally {
                    httpResponse2.close();
                    client2.close();
                }
            }

        } catch (IOException e) {
            e.printStackTrace();
        } finally {
            try {
                // close HttpClient and release all system resources
                client.close();
            } catch (IOException e) {
                e.printStackTrace();
            }
        }
    }

    private static String printResponse(HttpResponse httpResponse, CookieStore cookieStore)
            throws ParseException, IOException {
        HttpEntity entity = httpResponse.getEntity();
        // response status code
        System.out.println("status:" + httpResponse.getStatusLine());
        System.out.println("headers:");
        HeaderIterator iterator = httpResponse.headerIterator();
        while (iterator.hasNext()) {
            System.out.println("\t" + iterator.next());
        }

        System.out.println("cookies:");
        List<Cookie> cookies = cookieStore.getCookies();
        for (int i = 0; i < cookies.size(); i++) {
            System.out.println("\t" + cookies.get(i));
        }
        // 判斷 response entity 是否 null
        String responseString = null;
        if (entity != null) {
            responseString = EntityUtils.toString(entity);
            System.out.println("response length:" + responseString.length());
            System.out.println("response content:"
                    + responseString.replace("\r\n", ""));
        }

        return responseString;
    }

    private static List<NameValuePair> getParam(Map parameterMap) {
        List<NameValuePair> param = new ArrayList<NameValuePair>();
        Iterator it = parameterMap.entrySet().iterator();
        while (it.hasNext()) {
            Entry parmEntry = (Entry) it.next();
            param.add(new BasicNameValuePair((String) parmEntry.getKey(),
                    (String) parmEntry.getValue()));
        }
        return param;
    }
}

  • 使用 CookieStore

修改 TestHttpContext,利用既有的 cookieStore 產生新的 CloseableHttpClient: CloseableHttpClient client2 = HttpClients.custom().setDefaultCookieStore(cookieStore).build();


    if( !loginFailedFlag ) {
        // 以 cookieStore, 建立新的 CloseableHttpClient
        CloseableHttpClient client2 = HttpClients.custom()
                .setDefaultCookieStore(cookieStore).build();

        // 執行get請求
        HttpGet httpGet = new HttpGet(testUrl);
        System.out.println("request line:" + httpGet.getRequestLine());
        CloseableHttpResponse httpResponse2 = client2.execute(httpGet);

        try {
            printResponse(httpResponse2, cookieStore);
        } finally {
            httpResponse2.close();
            client2.close();
        }
    }
```

### HTTP Protocol Interceptors

可在處理 http message 時,加上一些特定的 Header,也可以在 outgoing message 中加上特別的 header,或是進行 content 壓縮/解壓縮,通常是用 "Decorator" pattern 實作的。

interceptor 可透過 context 共享資訊,例如在連續多個 request 中儲存 processing state。

protocol interceptor 必須要實作為 thread-safe,除非有將變數 synchronized,否則不要使用 instance variable。

public class TestHttpInterceptors {


public static void main(String[] args) {
    TestHttpInterceptors test = new TestHttpInterceptors();

    try {
        test.testInterceptors();

    } catch (Exception e) {
        e.printStackTrace();
    }

}

public void testInterceptors() throws IOException {
    final HttpClientContext httpClientContext = HttpClientContext.create();

    AtomicInteger count = new AtomicInteger(1);
    httpClientContext.setAttribute("Count", count);

    // request interceptor
    HttpRequestInterceptor httpRequestInterceptor = new HttpRequestInterceptor() {
        public void process(HttpRequest httpRequest, HttpContext httpContext) throws HttpException, IOException {
            AtomicInteger count = (AtomicInteger) httpContext.getAttribute("Count");

            httpRequest.addHeader("Count", String.valueOf(count.getAndIncrement()));
        }
    };

    // response handler
    ResponseHandler<String> responseHandler = new ResponseHandler<String>() {
        public String handleResponse(HttpResponse httpResponse) throws ClientProtocolException, IOException {

// HeaderIterator iterator = httpResponse.headerIterator();
// while (iterator.hasNext()) {
// System.out.println("\t" + iterator.next());
// }


            HttpEntity entity = httpResponse.getEntity();
            if (entity != null) {
                return EntityUtils.toString(entity);
            }
            return null;
        }
    };

    final CloseableHttpClient httpClient = HttpClients
            .custom()
            .addInterceptorLast(httpRequestInterceptor)
            .build();

    final HttpGet httpget = new HttpGet("http://192.168.1.24/");

    for (int i = 0; i < 20; i++) {

        String result = httpClient.execute(httpget, responseHandler, httpClientContext);

// System.out.println(result);
}


}

}
```


Exception Handling


HTTP Protocol processor 會產生兩種 Exceptions: java.io.IOException (socket timeout, socket reset) 及 HttpException (HTTP failure)。HttpClient 會 re-throw HttpException 為 ClientProtocolExcpetion (subclass of java.io.IOException),因此我們只需要 catch IOException,就可同時處理兩種錯誤狀況。


HTTP protocol 是一種簡單的 request/response protocol,沒有 transaction processing 的功能。預設 HttpClient 會自動由 I/O Exception 恢復。


  1. HttpClient不會嘗試從任何邏輯或HTTP協議錯誤中恢復(繼承自HttpException class)

  2. HttpClient將自動重試被認定的冪等方法

  3. HttpClient將自動重試當HTTP請求仍然在傳送到目標服務器,但卻失敗的方法(例如請求還沒有完全傳輸到服務器)


HttpRequestRetryHandler myRetryHandler = new HttpRequestRetryHandler(){
    public boolean retryRequest(IOException exception, int executionCouont, HttpContext context){
        if(executionCount >= 5){
            return false;
        }
        if(exception instanceof InterruptedIOException){
            return false;
        }
        if(exception instanceof UnknownHostException){
            return false;
        }
        if(exception instanceof ConnecTimeoutException){
            return false;
        }
        if(exception instanceof SSLException){
            return false;
        }
        HttpClientContext clientContext = HttpClientContext.adapt(context);
        HttpRequest request = clientContext.getRequest();
        boolean idmpotent - !(request instanceof HttpEntityEnclosingRequest);
        if(idempotent){
            return true;
        }
        return false;
    }
};
CloseableHttpClient httpclient = HttpClients.custom().setRetryHandler(myRetryHandler).build();

Aborting Requests


可以在執行的任何階段呼叫 HttpUriRequest#abort() 方法終止 request,提前終止該 request 並解除執行線程對I/O操作的阻塞。該方法是 thread-safe,可以從任何thread 呼叫該 method,如果HTTP請求終止,會拋出InterruptedIOException。


Redirect Handling


HttpClient自動處理所有類型的 Redirect,除了那些HTTP spec 要求必須用戶介入的狀況,POST 和 PUT 的 see Other(狀態code 303)重定向按HTTP規範的要求轉換成GET請求。可以自定義重定向策略覆蓋HTTP規範規定的方式.


LaxRedirectStrategy redirectStrategy = new LaxRedirectStrategy();
CloseableHttpClient httpclient = HttpClients.custom()
.setRedirectStrategy(redirectStrategy)
.build();

HttpClient經常需要在執行過程中重寫請求信息,默認的HTTP/1.0和HTTP/1.1通常使用相對請求URIs,原始的請求也可能從其他位置重定向多次,最終的絕對HTTP位置可使用原始的 request 和 context 獲得。URIUtils#resolve可以用來解釋絕對URI用於最終的 request,該方法包括重定向請求或原始請求的最後一個片段的 identifier。


CloseableHttpClient httpclient = HttpClients.createDefault();
HttpClientContext context = HttpClientContext.create();
HttpGet httpget = new HttpGet();
CloseableHttpResponse response = httpclient.execute(httpget,context);
try{
    HttpPost target = context.getTargetHost();
    List<URI> redirectLocations = context.getRedirectLocations();
    URI location = URIUtils.resolve(httpget.getURI(),target,redirectLocations);
    System.out.println("Final HTTP location: " + location.toASCIIString());
} finally {
    response.close();
}

References


使用 httpclient 連接池及注意事項


HttpClient 4 Cookbook


Posting with HttpClient


HttpClient tutorial


HTTP context的使用


HttpClient4.x 使用cookie保持會話


Java爬蟲入門簡介(三)——HttpClient保存、使用Cookie請求


HttpClient獲取Cookie的一次踩坑實錄


Apache HttpClient 4.5 How to Get Server Certificates

2018/2/5

Pacemaker & Corosync


Pacemaker 作為一個 cluster resource manager,負責處理多個 server node 旗下軟體的生命週期,他是透過 cluster services 監控及復原 node 的狀態,cluster service 提供 messaging 與 membership 管理機制,常見的 cluster service 有 corosync, cman及 heartbeat。


以往在處理cluster service 是用 heartbeat,但在 v3 以後,該專案拆分為多個部分,包含 Cluster Glue、Resource Agents、messaging layer(Heartbeat proper)、Local Resource Manager,以及 Cluster Reource Manager,而pacemaker 就是拆分出來的 resource manager,而新版的 heartbeat 只負責處理各 server node 之間的 messaging。


Pacemaker 主要功能包含


  1. server node 及 service 的故障檢測和恢復
  2. 多樣化的 storage,不需要 shared storage
  3. 多樣化的 resources,任何可以寫成 script 的服務都可以被 clustered
  4. 支援 fencing (STONITH),確保 data integrity
  5. 同時支持多種集群配置模式,規模大或小都可以
  6. 同時支援 quorate 以及 resource-driven 兩種 clusters
  7. 支援多種 redundancy configuration
  8. 自動化 replicated configuration,可由任意一個 node 更新 config
  9. 可指定 cluster-wide service ordering, colocation 及 anti-colocation
  10. 支援進階的 service types: (1) clones: 用在需要在多個 nodes 啟動的 services (2) multi-state: 用在 master/slave, primary/secondary
  11. unified, scriptable cluster management tools

STONITH: Shoot-The-Other-Node-In-The-Head 的縮寫,就是將發生問題的 node 關掉的功能,通常試用 remote power switch 來實現。


High-availability cluster: Node Configurations 中提到,最常見的兩個 server node 的 cluster 架構如下



如果架構牽涉到多個 nodes,則有下列的情況


  1. Active/Active


    要導向到 failed node 的 traffic,會轉送到其他 active nodes,這只能用在所有 nodes 都使用相同的 software configuration 的情況

  2. Active/Passive


    每個 node 都完整提供 redundant instance,備援節點只會在 primary node failed 時,切換為 online,這種架構需要增加 hardware


  3. N+1


    提供一個單一的 extra node,會在某個 node failed 時,接手該 node 的工作,切換為 on-line,每個 node 會有不同的 software configuration,該 extra node 要能夠替代其他 nodes 的配置。當 N 為 1,就等同於 Active/Passive 的架構。


  4. N+M


    如果這個 cluster 提供了多個 services,單一個 failover node 不敷使用,這時需要多個 standby nodes

  5. N-to-1


    可讓 failover node 暫時變為 active node,直到原本的 node 已經復原並 on-line,而服務會再切換回原本的 service node。

  6. N-to-N


    合併了 active/active 及 N+M 的概念,當發生 failed node,會將 traffic 導向到其他的 active nodes,不需要 standby node,但需要所有 active nodes 都有接手其他 nodes service 的能力。


  7. split-site


    多個機房的 clustering



note: OpenAIS 是對 Service Availability Forum 的AIS (Application Interface Specification) 的實作,包含了 node 管理, messaging, monitoring 等功能,但沒有 cluster resource manager 的功能,因此需要使用 pacemaker 或 rgmanager 作為 resource manager。Corosync Cluster Engine 就是由 OpenAIS 發展而來的。


Sample: Apache httpd Active-Passive cluster


以 vagrant 準備兩個 VM: web1, web2,再根據
How To Set Up an Apache Active-Passive Cluster Using Pacemaker on CentOS 7 的說明,測試設定 web1 及 web2 為 Apache httpd Active-Passive cluster 架構。


Vagrant.configure("2") do |config|
  config.vm.provision "shell", inline: "echo Hello"

  config.vm.define "web1" do |web1|
    web1.vm.box = "geerlingguy/centos7"
    web1.vm.hostname = "web1"

    web1.vm.network "private_network", ip: "192.168.0.100"
    web1.vm.network "public_network", ip: "192.168.1.24", bridge: "en0: 乙太網路", auto_config: false

    web1.vm.provision "shell",
        run: "always",
        inline: "route add default gw 192.168.1.1"
  end

  config.vm.define "web2" do |web2|
    web2.vm.box = "geerlingguy/centos7"
    web2.vm.hostname = "web2"

    web2.vm.network "private_network", ip: "192.168.0.200"
    web2.vm.network "public_network", ip: "192.168.1.25", bridge: "en0: 乙太網路", auto_config: false

    web2.vm.provision "shell",
        run: "always",
        inline: "route add default gw 192.168.1.1"
  end
end

編輯 /etc/hosts,分別讓兩台機器都能以 hostname 連接到對方


$ vi /etc/hosts

192.168.0.100       web1
192.168.0.200       web2

安裝 apache httpd


yum -y install httpd

修改 status page


$ vi /etc/httpd/conf.d/status.conf

<Location /server-status>
   SetHandler server-status
   Order Deny,Allow
   Deny from all
   Allow from 127.0.0.1
</Location>

分別在兩台機器,製作不同的首頁


$ cat <<-END > /var/www/html/index.html
<html>
<body>hello web1</body>
</html>

END

$ cat <<-END > /var/www/html/index.html
<html>
<body>hello web2</body>
</html>

END



安裝 pacemaker,安裝後會產生新的帳號 hacluster


yum -y install pacemaker pcs

systemctl enable pcsd.service
systemctl start pcsd.service

設定兩台機器相同的 hacluster 密碼


sudo passwd hacluster



設定 pacemaker


檢查 firewall status,如果沒有啟動,就啟動 firewalld


firewall-cmd --state

systemctl start firewalld.service

在 firewalld 新增一個 high-availability service


firewall-cmd --permanent --add-service=high-availability

# reload firewalld
firewall-cmd --reload

同時在兩台機器將 pacemaker 及 corosync 都設定為開機啟動


systemctl enable corosync.service
systemctl enable pacemaker.service

因為這兩台機器已經都安裝且設定了 pacemaker,接下來,我們只需要在其中一台機器設定 authentication


$ pcs cluster auth web1 web2
Username: hacluster
Password:
web2: Authorized
web1: Authorized

產生同步的 corosync 設定


$ sudo pcs cluster setup --name webcluster web1 web2

Destroying cluster on nodes: web1, web2...
web1: Stopping Cluster (pacemaker)...
web2: Stopping Cluster (pacemaker)...
web1: Successfully destroyed cluster
web2: Successfully destroyed cluster

Sending 'pacemaker_remote authkey' to 'web1', 'web2'
web1: successful distribution of the file 'pacemaker_remote authkey'
web2: successful distribution of the file 'pacemaker_remote authkey'
Sending cluster config files to the nodes...
web1: Succeeded
web2: Succeeded

Synchronizing pcsd certificates on nodes web1, web2...
web2: Success
web1: Success
Restarting pcsd on the nodes in order to reload the certificates...
web2: Success
web1: Success

接下來就可以看到,剛剛設定的 webcluster 已經寫入這個設定檔 /etc/corosync/corosync.conf


# more corosync.conf
totem {
    version: 2
    secauth: off
    cluster_name: webcluster
    transport: udpu
}

nodelist {
    node {
        ring0_addr: web1
        nodeid: 1
    }

    node {
        ring0_addr: web2
        nodeid: 2
    }
}

quorum {
    provider: corosync_votequorum
    two_node: 1
}

logging {
    to_logfile: yes
    logfile: /var/log/cluster/corosync.log
    to_syslog: yes
}



啟動 Cluster


pcs cluster start --all

檢查 cluster 狀態


# pcs status
Cluster name: webcluster
WARNING: no stonith devices and stonith-enabled is not false
Stack: unknown
Current DC: NONE
Last updated: Mon Dec 18 07:39:34 2017
Last change: Mon Dec 18 07:39:20 2017 by hacluster via crmd on web2

2 nodes configured
0 resources configured

Node web1: UNCLEAN (offline)
Online: [ web2 ]

No resources


Daemon Status:
  corosync: active/disabled
  pacemaker: active/disabled
  pcsd: active/enabled

Note: 發生 pacemaker node is UNCLEAN (offline) 的問題,這必須要修改 /etc/hosts


分別修改 /etc/hosts 將 127.0.0.1 web1 及 web2,這一行刪除,並重新啟動 corosync


#127.0.0.1  web1    web1

systemctl restart corosync.service

接下來就可以看到 pcs 正常的狀態


# pcs status
Cluster name: webcluster
WARNING: no stonith devices and stonith-enabled is not false
Stack: corosync
Current DC: web1 (version 1.1.16-12.el7_4.5-94ff4df) - partition with quorum
Last updated: Mon Dec 18 07:52:36 2017
Last change: Mon Dec 18 07:45:38 2017 by hacluster via crmd on web2

2 nodes configured
0 resources configured

Online: [ web1 web2 ]

No resources


Daemon Status:
  corosync: active/enabled
  pacemaker: active/enabled
  pcsd: active/enabled



在 pcs status 看到的 STONITH(Shoot-The-Other-Node-In-The-Head) warning,可以將 stonith 關閉解決


pcs property set stonith-enabled=false

在一半以上的 nodes online 時,cluster 會產生 quorum,Pacemaker 預設是在沒有 quorum 時,就會關閉所有 resources,因為現在是以兩台機器進行測試,因此要關閉 quorum 的功能。


pcs property set no-quorum-policy=ignore



設定 Virtual IP


pcs resource create Cluster_VIP ocf:heartbeat:IPaddr2 ip=192.168.1.26 cidr_netmask=24 op monitor interval=20s

查詢 ip addr,可發現目前 web1 有兩個 Public IPs: 192.168.1.24 及 192.168.1.26


# ip addr show

4: enp0s9: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 08:00:27:c4:2b:2d brd ff:ff:ff:ff:ff:ff
    inet 192.168.1.24/24 brd 192.168.1.255 scope global enp0s9
       valid_lft forever preferred_lft forever
    inet 192.168.1.26/24 brd 192.168.1.255 scope global secondary enp0s9
       valid_lft forever preferred_lft forever
    inet6 fe80::a00:27ff:fec4:2b2d/64 scope link
       valid_lft forever preferred_lft forever

# pcs status

.....

Full list of resources:

 Cluster_VIP    (ocf::heartbeat:IPaddr2):   Started web1



將 Apache httpd 加入 cluster resource,resource agent 為 ocf:heartbeat:apache


pcs resource create WebServer ocf:heartbeat:apache configfile=/etc/httpd/conf/httpd.conf statusurl="http://127.0.0.1/server-status" op monitor interval=20s



要確保兩個 resource 運作在同一台機器有兩種方式


  1. 將 ClusterVIP 及 WebServer 綁定為同一個 resource group,並設定 ClusterVIP 啟動順序先於 WebServer

pcs resource group add WebGroup Cluster_VIP
pcs resource group add WebGroup WebServer

pcs constraint order start Cluster_VIP then start WebServer

  1. 設定 colocation constraint

pcs constraint colocation add WebServer Cluster_VIP INFINITY



測試 cluster,首先以 browser 瀏覽 Virtual IP 首頁http://192.168.1.26,畫面上會看到 hello web1


將 web1 關機


vagrant halt web1

Virtual IP 首頁http://192.168.1.26,畫面上會看到 hello web2


將 web1 啟動


vagrant up web1

這時還是維持在 web2,除非再把 web2 關機,服務就會回到 web1




如果希望盡量以 web1 為主,web2 為輔,當 web1 開機時,就使用 web1,必須要增加 location 限制,將 web1 的priority 調高。當 web1 offline 而 web2 online,如果 web1 online 了,網頁服務還是會回到 web1。


pcs constraint location WebServer prefers web1=50
pcs constraint location WebServer prefers web2=45

References


How To Create a High Availability Setup with Corosync, Pacemaker, and Floating IPs on Ubuntu 14.04


將 Heartbeat 換成 Pacemaker+Corosync


High Availability and Pacemaker 101!


Automating Failover with Corosync and Pacemaker


透過 PACEMAKER 來配置 REDHAT 6 HIGH AVAILABILITY ADD-ON


CentOS7 架設 RHCS (High-Availability Server)


Pacemaker + Corosync 做服務 HA


How To Set Up an Apache Active-Passive Cluster Using Pacemaker on CentOS 7


在 CentOS7/RHEL7 上,學習架設 High-Availability 服務(一)


corosync+pacemaker 高可用集群


Centos7之pacemaker高可用安裝配置詳解


Linux 高可用(HA)集群之Pacemaker詳解


高可用centos7 HA:corosync+packmaker+http\mysql


使用 Load Balancer,Corosync,Pacemaker 搭建 Linux 高可用集群


CentOS 7 で DRBD/Pacemaker/Corosync で High Availability NFS


在 CentOS 7 上使用 PaceMaker 構建 NFS HA 服務


Corosync+pacemaker+DRBD+mysql(mariadb)實現高可用(ha)的mysql集群(centos7)

2018/1/29

Vagrant


Vagrant 是一個管理 Virtual Machine 的 command line utility 工具。最初支援 VirtualBox,但現在也支援了 docker、Parallels、VMWare。


安裝


先安裝 VirtualBox: Download VirtualBox


然後在 Download Vagrant,根據不同 OS 下載 binary 套件直接安裝就好了。


vagrant CLI


vagrant 本身是一個 CLI tool,目的是為了能在 script 直接操作並使用 VM,能夠讓系統安裝及測試自動化。


如果需要一個的 CentOS 7 VM 可以用以下指令處理,vagrant 有個集中分享的 Vagrant Cloud repository,我們可以直接搜尋centos7 找到適合與最多人使用的 VM。


我們使用 geerlingguy/centos7 Vagrant box VM


mkdir vagrant_centos7
cd vagrant_centos7

# 以 init 指令下載 centos 7 Vagrantfile
vagrant init geerlingguy/centos7

# 啟動 VM
vagrant up

# VM 關機
vagrant halt

# 刪除 VM
vagrant destroy

# 查詢 VM 狀態
vagrant status

# 查看 ssh 資訊
vagrant ssh-config

其他常用的指令


# 列出所有 vagrant VMs
vagrant box list

# guest OS 及 host OS 的 port 對應表
vagrant port
    22 (guest) => 2222 (host)

# 以 ssh 登入 VM
vagrant ssh

Vagrantfile


剛剛使用的 CentOS 7 VM Vagranfile source code 也是透過定義 Vagrantfile 再分享到 Cloud。


而我們的 Vagrantfile 只需要這樣的設定,就能夠引用 geerlingguy 的 centos7


# -*- mode: ruby -*-
# vi: set ft=ruby :

Vagrant.configure("2") do |config|
  config.vm.box = "geerlingguy/centos7"
end

被引用的 Vagratfile 定義會被下載到 ~/.vagrant.d 這個目錄中,可在 boxes 目錄找到相關資訊


~/.vagrant.d/boxes/geerlingguy-VAGRANTSLASH-centos7

Vagrantfile 是採用 ruby 語法,當我們在某個目錄 (/Users/user/VirtualBoxVMs/vagrant_centos7/) 執行 vagrant 指令時,他會依照以下順序尋找 Vagrantfile


/Users/user/VirtualBoxVMs/vagrant_centos7/Vagrantfile
/Users/user/VirtualBoxVMs/Vagrantfile
/Users/user/Vagrantfile
/Users/Vagrantfile
/Vagrantfile



  • Configuration Version

Vagrant.configure 後面的數字,代表不同版本的 vagrant 語法,


  1. "1" 表示為 1.0.x 的語法
  2. "2" 表示為 1.1+ 到 2.0 版的語法

Vagrant.configure("2") do |config|
  # ...
end



  • Networking

VM 的網路設定決定了 VM 跟 host machine 的溝通介面,網路部分分為三種,Forwarded Ports、Private Network及 Public Network。


Forwarded Ports: forwarded_port


可開放 guest machine 的某個 Port,並轉換到 host machine 的另一個 Port,TCP 或 UDP 都可以。


guest machine 啟動了一個 web server,運作在 TCP Port 80,可 forward 到 host machine 的 TCP 8080


config.vm.network "forwarded_port", guest: 80, host: 8080

跟上面一樣,將 TCP 80 forward 到 TCP 8080,同時限制只能用 127.0.0.1 存取 8080 這個 forwarded port 設定,也可以加上 TCP/UDP 的 protocol 限制,預設為 TCP。


config.vm.network "forwarded_port", guest: 80, host: 8080, host_ip: "127.0.0.1"

config.vm.network "forwarded_port", guest: 80, host: 8080, host_ip: "127.0.0.1", protocol: "tcp"

Private Network


可透過 Internet 無法使用的 IP 存取 guest machine,在 VM 的網路設定通常稱為 NAT


  1. DHCP


    config.vm.network "private_network", type: "dhcp"
  2. Static IP


    config.vm.network "private_network", ip: "192.168.50.4"
  3. IPv6


config.vm.network "private_network", ip: "fde4:8dba:82e1::c4"

# 加上 netmask
config.vm.network "private_network",
    ip: "fde4:8dba:82e1::c4",
    netmask: "96"

Public Network


在 VM 的網路設定通常稱為 Bridge network,就是讓 VM 直接取得可讓其他機器存取的網路設定。


  1. DHCP


    config.vm.network "public_network"
    
    # 使用 DHCP 的設定作為 default route
    config.vm.network "public_network",
      use_dhcp_assigned_default_route: true
  2. Static IP


    config.vm.network "public_network", ip: "192.168.0.17"

    如果 host machine 有多個網路介面,在啟動 vagrant VM 時,會詢問要使用哪一個網路介面,可以在設定 Public Netork 時,直接決定是用哪一個網路介面。


config.vm.network "public_network", bridge: "en1: Wi-Fi (AirPort)"

# 可指定多個網路介面
config.vm.network "public_network", bridge: [
  "en1: Wi-Fi (AirPort)",
  "en6: Broadcom NetXtreme Gigabit Ethernet Controller",
]

可利用 shell 指令設定 ip


Vagrant.configure("2") do |config|
  config.vm.network "public_network", auto_config: false

  # manual ip
  config.vm.provision "shell",
    run: "always",
    inline: "ifconfig eth1 192.168.0.17 netmask 255.255.255.0 up"

  # manual ipv6
  config.vm.provision "shell",
    run: "always",
    inline: "ifconfig eth1 inet6 add fc00::17/7"
end

設定固定 IP 以及 default route


Vagrant.configure("2") do |config|
  config.vm.network "public_network", ip: "192.168.0.17"

  # default router
  config.vm.provision "shell",
    run: "always",
    inline: "route add default gw 192.168.0.1"

  # default router ipv6
  config.vm.provision "shell",
    run: "always",
    inline: "route -A inet6 add default gw fc00::1 eth1"

  # delete default gw on eth0
  config.vm.provision "shell",
    run: "always",
    inline: "eval `route -n | awk '{ if ($8 ==\"eth0\" && $2 != \"0.0.0.0\") print \"route del default gw \" $2; }'`"
end



  • Synced Folders

可在 guest 及 host machine 之間共用資料夾,vagrant 預設會分享 project 目錄,也就是存放 Vagrantfile 的目錄到 /vagrant。


# 前面是 host machine folder,後面是 guest machine path
config.vm.synced_folder "src/", "/srv/website"

# disable default /vagrant shared folder
config.vm.synced_folder ".", "/vagrant", disabled: true

# 修改 folder owner, group
config.vm.synced_folder "src/", "/srv/website",
  owner: "root", group: "root"

也可以使用 NFSRSync或是 SMB 這三種 protocol




  • Provisioning

這是在啟動 VM 時,自動安裝軟體、修改設定的功能。當 VM box 需要微調時,可以利用這個功能進行調整。Provisioning 可使用 shell script, ansible, chef, puppet, salt, docker 等指令,比較基本的是直接用 shell scripts


Provisioning 會在這三個時間點發生作用


  1. 第一次以 vagrant up 啟動 VM,Provisioning 會有作用,但如果是已經啟動過的 VM,就不會執行 provisioning,但可用 --provision 強制執行。

  2. 在 VM 運作中,執行 vagrant provision

  3. 執行 vagrant reload --provision




Vagrant Shell provisioner 可在 guest machine 執行某個 script。


Inline Scripts


直接在 Vagrantfile 撰寫 scipt command


Vagrant.configure("2") do |config|
  config.vm.provision "shell",
    inline: "echo Hello, World"
end

在上面定義 script,然後在 config.vm.provision 中執行


$script = <<SCRIPT
echo I am provisioning...
date > /etc/vagrant_provisioned_at
SCRIPT

Vagrant.configure("2") do |config|
  config.vm.provision "shell", inline: $script
end

External Script


可執行 host machine 的某一個 shell script,也可以用某個網址下載 script


Vagrant.configure("2") do |config|
  config.vm.provision "shell", path: "script.sh"
end

Vagrant.configure("2") do |config|
  config.vm.provision "shell", path: "https://example.com/provisioner.sh"
end

如果要執行 guest machine 的 script


Vagrant.configure("2") do |config|
  config.vm.provision "shell",
    inline: "/bin/sh /path/to/the/script/already/on/the/guest.sh"
end

Script Arguments


利用 args 指定 script 參數


Vagrant.configure("2") do |config|
  config.vm.provision "shell" do |s|
    s.inline = "echo $1"
    s.args   = "'hello, world!'"
  end
end

Vagrant.configure("2") do |config|
  config.vm.provision "shell" do |s|
    s.inline = "echo $1"
    s.args   = ["hello, world!"]
  end
end



  • Multi-Machines

Multi-Machines 功能可在一個 Vagrantfile 中定義多個 guest machines。


定義兩個機器,一台是 web,一台是 db


Vagrant.configure("2") do |config|
  config.vm.provision "shell", inline: "echo Hello"

  config.vm.define "web" do |web|
    web.vm.box = "apache"
  end

  config.vm.define "db" do |db|
    db.vm.box = "mysql"
  end
end

執行 vagrant up 會同啟動 web 及 db,也可以用 vagrant up web 只啟動 web


Reference


Vagrant Tutorial(1)雲端研發人員,你也需要虛擬機!


Vagrant Tutorial(2)跟著流浪漢把玩虛擬機


Vagrant Tutorial(3)細說虛擬機生滅狀態


Vagrant Tutorial(4)虛擬機,若即若離的國中之國


Vagrant Tutorial(5)客製化虛擬機內容的幾種方法


使用Vagrant進行伺服器環境部署

2018/1/22

Ansible


目前在 DevOps 設定管理的工具中,以 Puppe、Chef、Salt、Ansible 四者最有名,而 Puppe 跟 Chef 兩者是以 ruby 開發, Salt 與 Ansible 都是以 python 開發的,在 20162017 DevOps 的統計中,Ansible 自 2015 由 10% 上升至 20% 及 21%,chef 及 puppet 在 2017 的比例有下降一些,但還是比較多人使用的工具。


Ansible is Simple IT Automation 最簡單的 IT 自動化工具,包含 自動化部署APP、自動化管理配置、自動化的持續交付、自動化的(AWS)雲服務管理。ansible 是利用 paramiko 開發的,paramiko是一個純Python實現的ssh協議庫。因此不需要在遠端主機上安裝client或agents,因為 ansible 單純地是以 ssh 來和遠程主機進行通訊。


在 Ender's Game 裡面,Ender 是利用一個的超光速即時的通訊系統,在後端指揮中心,遠端下達指令給前方的戰機及戰艦,這個系統被稱為 Ansible,用途是 instantaneous communication across any distance。因此 Ansible 就跟 devops 的目標一致,這樣的工具就是要讓隱身在後面的工程師,個個都像 Ender 一樣,一呼百應而且沒有絲毫的遲疑,當然瞬間的成敗也由 Ender 獨力承擔。


安裝


ansible 對於管理端的主機,稱為 control machine,必須要安裝 python 2.6+,被管理(託管)的主機,稱為 managed node,要安裝 sshd,也要安裝 python 2.6+。



在 macos 安裝 ansible 可使用 macports 或是 homebrew


sudo port install ansible

當然也可以因應不同 OS 用 yum, apt, pip, deb 進行安裝


安裝測試環境


可用 VM 來進行測試,目前可選用 virtualbox 或是 docker


  • vagrant

必須安裝 vagrant 及 virtualbox,可選用 vagrant 官方提供的 VM


mkdir vagrant_centos7
cd vagrant_centos7

vagrant init geerlingguy/centos7
vagrant up

# 關機
vagrant halt

# 重開機
vagrant reload

# ssh
vagrant ssh

# 移除
vagrant destory

取得 VM 的 ssh 設定


$ vagrant ssh-config
Host default
  HostName 127.0.0.1
  User vagrant
  Port 2222
  UserKnownHostsFile /dev/null
  StrictHostKeyChecking no
  PasswordAuthentication no
  IdentityFile /Users/charley/VirtualBoxVMs/vagrant/.vagrant/machines/default/virtualbox/private_key
  IdentitiesOnly yes
  LogLevel FATAL

ansible.cfg 設定檔


$ vi ansible.cfg
[defaults]

inventory = hosts
remote_user = vagrant
private_key_file = .vagrant/machines/default/virtualbox/private_key
host_key_checking = False

ansible 會依照這四個順序尋找適合的 ansible.cfg


* ANSIBLE_CONFIG 環境變數
* 目前工作目錄的 ansible.cfg
* 使用者 Home 目錄的 .ansible.cfg
* 系統設定 /etc/ansible/ansible.cfg (如果是用macports 安裝則是在 /opt/local/etc/ansible/ansible.cfg)

hosts 設定檔


$ vi hosts
server1  ansible_ssh_host=127.0.0.1  ansible_ssh_port=2222

[local]
server1

利用 ansible echo 一段文字


$ ansible all -m command -a 'echo Hello World on Vagrant.'
server1 | SUCCESS | rc=0 >>
Hello World on Vagrant.

  • docker

準備一個有安裝了 sshd 的 docker VM,將 10022 對應到 TCP 22 (ssh),因為 CentOS 7 預設會安裝 python 2.7.5,就不需要處理 python 的安裝問題。


docker run -d \
 -p 10022:22\
 --sysctl net.ipv6.conf.all.disable_ipv6=1\
 -e "container=docker" --privileged=true -v /sys/fs/cgroup:/sys/fs/cgroup --name atest centosssh /usr/sbin/init

如果直接用 ssh client 連線


ssh root@localhost -p 10022

在另一個新的工作目錄,建立 ansible.cfg 設定檔


$ vi ansible.cfg
[defaults]

inventory = hosts
remote_user = root
host_key_checking = False

建立 hosts 設定檔


$ vi hosts
server1  ansible_ssh_host=127.0.0.1  ansible_ssh_port=10022 ansible_ssh_pass=max168kit

[local]
server1

測試 echo 指令


$ ansible all -m command -a 'echo Hello World on Docker.'
server1 | SUCCESS | rc=0 >>
Hello World on Docker.

使用 ansible


有兩種方式使用 ansible,分別是 ad-hoc command 及 playbook


  • ad-hoc command

一次只能使用一個指令,像是在 console mode,一次下達一個指令


$ ansible all -m ping
server1 | SUCCESS => {
    "changed": false,
    "ping": "pong"
}

$ ansible all -m command -a "echo Hello World"
server1 | SUCCESS | rc=0 >>
Hello World

  • playbook

像是 shell script 一樣,可組合多個指令,這是一份使用 YAML 格式製作的文件。通常一個 playbook 中會有多個 Play, Task, Module


  1. Play: 某個特定的工作,裡面會包含多個 Task

  2. Task: 為了實現 Play,每一個 Play 都會有一個 Task 列表,每一個 Task 在所有對應的主機上都執行完成後,才會進行下一個 Task

  3. Module: 每一個 Task 的目的是執行一個 Module,Module 也就是 ansible 提供的一些操作指令,可到 Module Index 文件查詢可以使用的 Modules


以下是一個簡單的 hello world playbook,裡面有一個 Play: say 'hello world',兩個 Tasks,第一個 Task 使用了 command 這個 Module,第二個 Task 使用 debug


vi hello_world.yml


---

- name: say 'hello world'
  hosts: all
  tasks:

    - name: echo 'hello world'
      command: echo 'hello world'
      register: result

    - name: print stdout
      debug:
        msg: "{{ result.stdout }}"

執行 playbook


$ ansible-playbook hello_world.yml

PLAY [say 'hello world'] ***************************************************************************

TASK [Gathering Facts] *****************************************************************************
ok: [server1]

TASK [echo 'hello world'] **************************************************************************
changed: [server1]

TASK [print stdout] ********************************************************************************
ok: [server1] => {
    "msg": "hello world"
}

PLAY RECAP *****************************************************************************************
server1                    : ok=3    changed=1    unreachable=0    failed=0

常用的 Modules



- name: install the latest version of Apache
  yum:
    name: httpd
    state: latest

- name: remove the Apache package
  yum:
    name: httpd
    state: absent

- name: upgrade all packages
  yum:
    name: '*'
    state: latest

- name: upgrade all packages, excluding kernel & foo related packages
  yum:
    name: '*'
    state: latest
    exclude: kernel*,foo*

- name: install the 'Development tools' package group
  yum:
    name: "@Development tools"
    state: present

  • command 遠端執行某個 shell command,不支援變數 > , >, |, ; 和 & ,如果需要這些功能,要改用 shell

- name: Reboot at now
  command: /sbin/shutdown -r now
  
- name: create .ssh directory
  command: mkdir .ssh creates=.ssh/

- name: cat /etc/passwd
  command: cat passwd
  args:
    chdir: /etc


- name: check files number
  shell: ls /home/docker/ | wc -l

- name: kill all python process
  shell: kill -9 $(ps aux | grep python | awk '{ print $2 }')

  • copy 將 local file 傳送到遠端機器

- name: copy ssh public key to remote node
  copy:
    src: files/id_rsa.pub
    dest: /home/docker/.ssh/authorized_keys
    owner: docker
    group: docker
    mode: 0644
    
- name: copy nginx vhost and backup the original
  copy:
    src: files/ironman.conf
    dest: /etc/nginx/sites-available/default
    owner: root
    group: root
    mode: 0644
    backup: yes

  • file 遠端建立和刪除檔案、目錄、links

- name: touch a file, and set the permissions
  file:
    path: /etc/motd
    state: touch
    mode: "u=rw,g=r,o=r"

- name: create a directory, and set the permissions
  file:
    path: /home/docker/.ssh/
    state: directory
    owner: docker
    mode: "700"

- name: create a symlink file
  file:
    src: /tmp
    dest: /home/docker/tmp
    state: link

  • lineinfile 可用正規表示式對檔案進行插入或取代文字,類似 sed

- name: remove sudo permission of docker
  lineinfile:
    dest: /etc/sudoers
    state: absent
    regexp: '^docker'

- name: set localhost as 127.0.0.1
  lineinfile:
    dest: /etc/hosts
    regexp: '^127\.0\.0\.1'
    line: '127.0.0.1 localhost'
    owner: root
    group: root
    mode: 0644


- name: start nginx service
  service:
    name: nginx
    state: started

- name: stop nginx service
  service:
    name: nginx
    state: stopped

- name: restart network service
  service:
    name: network
    state: restarted
    args: eth0    

  • stat 檢查檔案狀態

- name: check the 'vimrc' target exists
  stat:
    path: /home/docker/.vimrc
  register: stat_vimrc

- name: touch vimrc
  file:
    path: /home/docker/.vimrc
    state: touch
          mode: "u=rw,g=r,o=r"
  when: stat_vimrc.stat.exists == false

- name: Use md5sum to calculate checksum
  stat:
    path: /path/to/something
    checksum_algorithm: md5sum

References


Red Hat併購DevOps新秀Ansible


YAML wiki


Ansible 自動化部署工具


Chef 自動化部署工具


現代 IT 人一定要知道的 Ansible 自動化組態技巧


ansible Getting Started


Ansible中文權威指南


七分鐘掌握 Ansible 核心觀念


Ansible for Devops


自動化工具——ansible中文指南

2018/1/15

Typescript


TypeScript 是在 MS 工作的 Anders Hejlsberg (C#, TurboPascal 之父)提出的一個新的程式語言,不過他並不是一個無中生有的語言,TypeScript 是 JavaScript ES5 及 ES6 的 superset,可以跟既有的 JavaScript 程式完全相容,他主要是將若資料型別的 JavaScript 轉變為強資料型別的程式語言,在開發及編譯時,就能夠察覺一些程式語法的錯誤,同時增加物件導向的概念,它可以幫助 JavaScript 開發人員更容易撰寫及維護大規模的應用程式。


安裝


TypeScript 的編譯器 (tsc) 可透過 npm 安裝,另外 tsserver 是 node 執行檔,包含了 TypeScript 編譯器及 language service,介面為 JSON protocol,適用於 editors 及 IDE。


$ sudo npm install -g typescript
/opt/local/bin/tsc -> /opt/local/lib/node_modules/typescript/bin/tsc
/opt/local/bin/tsserver -> /opt/local/lib/node_modules/typescript/bin/tsserver
+ typescript@2.6.1
added 1 package in 2.927s

測試:建立一個新的 greeter.ts 檔案


function greeter(person) {
    return "Hello, " + person;
}

let user = "Jane User";

document.body.innerHTML = greeter(user);

透過 tsc 將 greeter.ts 編譯為 greeter.js


tsc greeter.ts

編譯後的 greeter.js 可在 console 用 nodejs 執行,或是放在一個網頁 greeter.html 裡面


$ node greeter.js
Hello, Jane User

greeter.html


<!DOCTYPE html>
<html>
    <head><title>TypeScript Greeter</title></head>
    <body>
        <script src="greeter.js"></script>
    </body>
</html>

開發的 IDE


雖然 Anders Hejlsberg 在官方網頁告訴我們要使用 Visual Studio plugin,但我們還是別的選擇



TypeScript Handbook


翻閱 TypeScript Handbook 手冊會發現,文件的編排方式,就像是在說明一個物件導向的程式語言一樣。


從基本的資料型別 Basic Types 開始,然後說明如何宣告變數,再來就是物件導向的核心: Interface, Classes 及 Functions,最後是 module 與 namespace。


Basic Types

Boolean, Number, String, Array, Tuple, Enum, Any, Void, Null and Undefined, never


// boolean
let isDone: boolean = false;

// number
let decimal: number = 6;
let hex: number = 0xf00d;
let binary: number = 0b1010;
let octal: number = 0o744;

// string  " 或是 ' 都可以
let color: string = "blue";
color = 'red';

特殊的 template strings `


let fullName: string = `Bob Bobbington`;
let age: number = 37;
let sentence: string = `Hello, my name is ${ fullName }.

I'll be ${ age + 1 } years old next month.`;

// 等同

let sentence: string = "Hello, my name is " + fullName + ".\n\n" +
    "I'll be " + (age + 1) + " years old next month.";

Array,有兩種宣告方式


let list: number[] = [1, 2, 3];

let list: Array<number> = [1, 2, 3];

Tuple: 就是固定 elements 個數的 array,且各元素的資料型別要相同


// Declare a tuple type
let x: [string, number];

// Initialize it
x = ["hello", 10]; // OK

// Initialize it incorrectly
// error TS2322: Type '[number, string]' is not assignable to type '[string, number]'. Type 'number' is not assignable to type 'string'.
x = [10, "hello"]; // Error

Enum


// 預設第一個元素,數字為 0
enum Color {Red, Green, Blue}
let c: Color = Color.Green;

let colorName: string = Color[2];
alert(colorName); // Displays 'Blue' as it's value is 2 above

// 可強制設定 enum numbers
enum Color {Red = 1, Green, Blue}
let c: Color = Color.Green;

enum Color {Red = 1, Green = 2, Blue = 4}
let c: Color = Color.Green;

Any: 宣告變數時,不知道這個變數的資料型別是什麼,可以任意變換自己的資料型別


let notSure: any = 4;
notSure = "maybe a string instead";
notSure = false; // okay, definitely a boolean

類似 Object 的功能,但 Object 只能讓我們指定 value,不能使用該 value 資料型別的任何一個 functions


let notSure: any = 4;
notSure.ifItExists(); // okay, ifItExists might exist at runtime
notSure.toFixed(); // okay, toFixed exists (but the compiler doesn't check)

let prettySure: Object = 4;
prettySure.toFixed(); // Error: Property 'toFixed' doesn't exist on type 'Object'

Void: 通常用在表示 function 沒有 return value


function warnUser(): void {
    alert("This is my warning message");
}

// 將變數宣告為 void,只能設定為 undefined 或是 null
let unusable: void = undefined;

Null and Undefined: 預設 null 及 undefined 是所有其他資料型別的 subtypes,換句話說,可以將 undefined 指定給 number,但如果編譯時加上 --strictNullChecks,就可以限制只能將 null 及 undefined 指定給 void


// Not much else we can assign to these variables!
let u: undefined = undefined;
let n: null = null;

Never: 代表 type of values that never occur,例如可以設定某個只會 throw exception 的 function 的 return value 為 never。他是獨立的,不是任何一種資料型別的 subtype,即使是 any 也不能指定給 never 型別的變數。


// Function returning never must have unreachable end point
function error(message: string): never {
    throw new Error(message);
}

// Inferred return type is never
function fail() {
    return error("Something failed");
}

// Function returning never must have unreachable end point
function infiniteLoop(): never {
    while (true) {
    }
}

Type assertions: 利用 compiler 檢查(確認) 資料型別,有兩種寫法 (someValue) 或是 (someValue as string)


let someValue: any = "this is a string";

let strLength: number = (<string>someValue).length;

let someValue2: any = "this is a string";

let strLength2: number = (someValue as string).length;

Variable Declaration

let 與 var 的差異


JavaScript Hositing: 在 JavaScript,變數可在使用後才被宣告,換句話說,變數可在宣告前,就使用它。在執行時期時,所有var變數都會自動被hoisting。如果程式中有參考到使用var定義過的變數時,會變成undefined,不會產生ERROR。


但 let 宣告的變數,不會被 Hositing,他只能作用在 { } 區塊範圍中。若程式中有參考到let定義過的變數時,因作用區塊不同會產生ERROR,此行為比較接近常用的程式語言寫法。


在 for 裡面,i 會持續被重新定義,所以在 setTimeout 後,最後使用的 i 都會是 5


for (var i = 0; i < 5 ; i++) {
    setTimeout(function() { console.log(i); }, 100 * i);
}

$ node greeter.js
5
5
5
5
5

let 的變數不會被重新宣告,能夠維持 i 的實際變數 value


for (let i = 0; i < 5 ; i++) {
    setTimeout(function() { console.log(i); }, 100 * i);
}

$ node greeter.js
0
1
2
3
4



const: 不能被 re-assign 的變數


const numLivesForCat = 9;
const kitty = {
    name: "Aurora",
    numLives: numLivesForCat,
}

// Error,以 const 宣告的 kitty 是一個有 readonly 屬性的object
kitty = {
    name: "Danielle",
    numLives: numLivesForCat
};

// all "okay"
kitty.name = "Rory";
kitty.name = "Kitty";
kitty.name = "Cat";
kitty.numLives--;

Functions

typescript 跟 javascript 一樣,可以建立 named 及 anonymous 兩種 functions


// Named function
function add(x, y) {
    return x + y;
}

// Anonymous function
let myAdd = function(x, y) { return x + y; };



爲 function 參數及 return value 加上 data type


function add(x: number, y: number): number {
    return x + y;
}

let myAdd = function(x: number, y: number): number { return x + y; };

增加 function type,下面的 (x: number, y: number) => number 就是 myAdd 的 function type,function type 裡面的變數名稱只是輔助使用幫助閱讀而已,實際上跟後面的程式本體沒有關係。


let myAdd: (x: number, y: number) => number =
    function(x: number, y: number): number { return x + y; };
    
let myAdd: (baseValue: number, increment: number) => number =
    function(x: number, y: number): number { return x + y; };



Optional 及 Default Parameters


function 的所有參數,在呼叫該 funciton 時,預設都是必要的。如果是 Optional 參數,要在定義時加上 ? ,也可以為參數寫上 default value


以下這兩個 function 的 function type 都是 (firstName: string, lastName?: string) => string


function buildName(firstName: string, lastName?: string) {
    if (lastName)
        return firstName + " " + lastName;
    else
        return firstName;
}

function buildName2(firstName: string, lastName = "Smith") {
    return firstName + " " + lastName;
}



Rest Parameters: ... 將剩下的參數集合為一個 array


function buildName(firstName: string, ...restOfName: string[]) {
    return firstName + " " + restOfName.join(" ");
}

let employeeName = buildName("Joseph", "Samuel", "Lucas", "MacKinzie");

let buildNameFun: (fname: string, ...rest: string[]) => string = buildName;



this: 在 javascript 的 function 被呼叫時,同時會設定 this 這個變數。但通常 function 會先被定義,而在後面才被呼叫,因此常常會弄錯 this 指定的對象。


Arrow function: ()=> 在 arrow function 中的 this,會指向該 function 定義時的 object,而不是使用該 function 時的 object


foo(x, y) => {
    x++; 
    y--; 
    return x+y;
}

沒有使用arrow function,在呼叫 says 時,setTimeout 裡面的 this 就不是 Animal 而是 window.this


//沒有使用arrow function

class Animal {
    constructor(){
        this.type = 'animal'
    }
    says(say){
        setTimeout(function(){
            console.log(this.type + ' says ' + say)
        }, 1000)
    }
}

var animal = new Animal()
animal.says('hi')  //undefined says hi

將 function() 改為 () =>


class Animal {
    constructor(){
        this.type = 'animal'
    }
    says(say){
        setTimeout( () => {
            console.log(this.type + ' says ' + say)
        }, 1000)
    }
}
var animal = new Animal()
animal.says('hi')  //animal says hi



Overloads: 因為 javascript 是非常動態的語言,常常會遇到某個 function 會在不同狀況,回傳不同資料型別的資料。


定義該 function 時會很直覺地將該 function 的 return value 定義為 any。但這樣卻失去了 TypeScript 的強資料型別的檢查功能。


解決方式是在前面明確地將 function 的各種參數及 return 的狀況都定義出來。


function pickCard(x: {suit: string; card: number; }[]): number;
function pickCard(x: number): {suit: string; card: number; };

let suits = ["hearts", "spades", "clubs", "diamonds"];

function pickCard(x: {suit: string; card: number; }[]): number;
function pickCard(x: number): {suit: string; card: number; };
function pickCard(x): any {
    // Check to see if we're working with an object/array
    // if so, they gave us the deck and we'll pick the card
    if (typeof x == "object") {
        let pickedCard = Math.floor(Math.random() * x.length);
        return pickedCard;
    }
    // Otherwise just let them pick the card
    else if (typeof x == "number") {
        let pickedSuit = Math.floor(x / 13);
        return { suit: suits[pickedSuit], card: x % 13 };
    }
}

let myDeck = [{ suit: "diamonds", card: 2 }, { suit: "spades", card: 10 }, { suit: "hearts", card: 4 }];
let pickedCard1 = myDeck[pickCard(myDeck)];
alert("card: " + pickedCard1.card + " of " + pickedCard1.suit);

let pickedCard2 = pickCard(15);
alert("card: " + pickedCard2.card + " of " + pickedCard2.suit);

Interfaces

interface 就跟 java interface 功能一樣,可提供 function 定義的檢查,限制 function 必須在實作時,遵循 interface 的定義。


例如 printLabel 需要一個參數 lavelledObj,且該參數要有 label 屬性。


function printLabel(labelledObj: { label: string }) {
    console.log(labelledObj.label);
}

let myObj = {size: 10, label: "Size 10 Object"};
printLabel(myObj);

增加一個 interface 定義,讓 labelledObj 定義為 LabelledValue 型別,就能在編譯時,檢查 myObj 是否有遵循 LabelledValue 的介面定義。


interface LabelledValue {
    label: string;
}

function printLabel(labelledObj: LabelledValue) {
    console.log(labelledObj.label);
}

let myObj = {size: 10, label: "Size 10 Object"};
printLabel(myObj);



Optional interface properties: interface 的屬性,可用 ? 代表該屬性可有可無


interface SquareConfig {
    color?: string;
    width?: number;
}

function createSquare(config: SquareConfig): {color: string; area: number} {
    let newSquare = {color: "white", area: 100};
    if (config.color) {
        newSquare.color = config.color;
    }
    if (config.width) {
        newSquare.area = config.width * config.width;
    }
    return newSquare;
}

let mySquare = createSquare({color: "black"});



readonly properties: 限制該屬性在 assign 後,就不能被修改


interface Point {
    readonly x: number;
    readonly y: number;
}

let p1: Point = { x: 10, y: 20 };
p1.x = 5; // error!

作用跟 const 很像,差別是 properties 是用 readonly,而 variables 是用 const




Function Types: 可利用 interface 描述 Function Types


interface SearchFunc {
    (source: string, subString: string): boolean;
}

let mySearch: SearchFunc;
mySearch = function(source: string, subString: string) {
    let result = source.search(subString);
    return result > -1;
}

因 compiler 的檢查機制,可以簡化 function 定義裡面原本要寫的 參數及 return value 的 data types


let mySearch: SearchFunc;
mySearch = function(src, sub) {
    let result = src.search(sub);
    return result > -1;
}



Indexable Types: have an index signature that describes the types we can use to index into the object


interface StringArray {
    [index: number]: string;
}

let myArray: StringArray;
myArray = ["Bob", "Fred"];

let myStr: string = myArray[0];



Class Types: 就跟 c#, java 的 interface 與 class 的關係一樣,class 可 implements interfaces。interface 可定義 properties 及 functions


interface ClockInterface {
    currentTime: Date;
    setTime(d: Date);
}

class Clock implements ClockInterface {
    currentTime: Date;
    setTime(d: Date) {
        this.currentTime = d;
    }
    constructor(h: number, m: number) { }
}



static 與 instance sides of classes 的差異


class 有兩種面向: static side 與 instance side


如果需要一個特殊的 constructor,以下是有問題的程式,因為當 class 實作 interface 時,只會檢查 instance side,但 constructor 是 static side。


interface ClockConstructor {
    new (hour: number, minute: number);
}

class Clock implements ClockConstructor {
    currentTime: Date;
    constructor(h: number, m: number) { }
}

必須將 static side 及 instance side 分成兩個 interfaces: ClockConstructor 是給 constructor 用的, ClockInterface 是 instance methods


interface ClockConstructor {
    new (hour: number, minute: number): ClockInterface;
}

interface ClockInterface {
    tick();
}

function createClock(ctor: ClockConstructor, hour: number, minute: number): ClockInterface {
    return new ctor(hour, minute);
}

class DigitalClock implements ClockInterface {
    constructor(h: number, m: number) { }
    tick() {
        console.log("beep beep");
    }
}
class AnalogClock implements ClockInterface {
    constructor(h: number, m: number) { }
    tick() {
        console.log("tick tock");
    }
}

let digital = createClock(DigitalClock, 12, 17);
digital.tick();
let analog = createClock(AnalogClock, 7, 32);
analog.tick();



Extending Interfaces: interfaces 可互相 extend,也就是可以從一個 interface 複製 members 到另一個


interface Shape {
    color: string;
}

interface PenStroke {
    penWidth: number;
}

interface Square extends Shape, PenStroke {
    sideLength: number;
}

let square = <Square>{};
square.color = "blue";
square.sideLength = 10;
square.penWidth = 5.0;

Classes

這是最簡單的 class,用 new 語法產生 instance


class Greeter {
    greeting: string;
    constructor(message: string) {
        this.greeting = message;
    }
    greet() {
        return "Hello, " + this.greeting;
    }
}

let greeter = new Greeter("world");



Inheritance: Dog 繼承 Animal,多了 move 這個 method


class Animal {
    move(distanceInMeters: number = 0) {
        console.log(`Animal moved ${distanceInMeters}m.`);
    }
}

class Dog extends Animal {
    bark() {
        console.log('Woof! Woof!');
    }
}

const dog = new Dog();
dog.bark();
dog.move(10);
dog.bark();



更複雜的例子 Animal: Horse and Snake,用 super 呼叫上層的 method


class Animal {
    name: string;
    constructor(theName: string) { this.name = theName; }
    move(distanceInMeters: number = 0) {
        console.log(`${this.name} moved ${distanceInMeters}m.`);
    }
}

class Snake extends Animal {
    constructor(name: string) { super(name); }
    move(distanceInMeters = 5) {
        console.log("Slithering...");
        super.move(distanceInMeters);
    }
}

class Horse extends Animal {
    constructor(name: string) { super(name); }
    move(distanceInMeters = 45) {
        console.log("Galloping...");
        super.move(distanceInMeters);
    }
}

let sam = new Snake("Sammy the Python");
let tom: Animal = new Horse("Tommy the Palomino");

sam.move();
tom.move(34);



public(預設), private, protected


private: 不能從 class 外面使用該 member
protected: 可在 subclass 使用該 member


class Person {
    protected name: string;
    constructor(name: string) { this.name = name; }
}

class Employee extends Person {
    private department: string;

    constructor(name: string, department: string) {
        super(name);
        this.department = department;
    }

    public getElevatorPitch() {
        return `Hello, my name is ${this.name} and I work in ${this.department}.`;
    }
}

let howard = new Employee("Howard", "Sales");
console.log(howard.getElevatorPitch());
// console.log(howard.name); // error

也可以將 constructor 設定為 protected,表示該 class 不能直接被 instantiated,但還是可以被繼承


class Person {
    protected name: string;
    protected constructor(theName: string) { this.name = theName; }
}

// Employee can extend Person
class Employee extends Person {
    private department: string;

    constructor(name: string, department: string) {
        super(name);
        this.department = department;
    }

    public getElevatorPitch() {
        return `Hello, my name is ${this.name} and I work in ${this.department}.`;
    }
}

let howard = new Employee("Howard", "Sales");
// let john = new Person("John"); // Error: The 'Person' constructor is protected



readonly: 可將 properties 設定為 readonly,但 readonly properties 必須在宣告或是 constructor 初始化


class Octopus {
    readonly name: string;
    readonly numberOfLegs: number = 8;
    constructor (theName: string) {
        this.name = theName;
    }
}
let dad = new Octopus("Man with the 8 strong legs");
// dad.name = "Man with the 3-piece suit"; // error! name is readonly.

可為 readonly 欄位加上 get set methods


let passcode = "secret passcode";

class Employee {
    private _fullName: string;

    get fullName(): string {
        return this._fullName;
    }

    set fullName(newName: string) {
        if (passcode && passcode == "secret passcode") {
            this._fullName = newName;
        }
        else {
            console.log("Error: Unauthorized update of employee!");
        }
    }
}

let employee = new Employee();
employee.fullName = "Bob Smith";
if (employee.fullName) {
    console.log(employee.fullName);
}

如果直接編譯會發生 error,可用 tsconfig.json 或是編譯的參數解決


error TS1056: Accessors are only available when targeting ECMAScript 5 and higher.

注意編譯時要加上 --target ES5


tsc --target ES5 greeter.ts



static properties: 直接用 Grid.origin 存取 origin


class Grid {
    static origin = {x: 0, y: 0};
    calculateDistanceFromOrigin(point: {x: number; y: number;}) {
        let xDist = (point.x - Grid.origin.x);
        let yDist = (point.y - Grid.origin.y);
        return Math.sqrt(xDist * xDist + yDist * yDist) / this.scale;
    }
    constructor (public scale: number) { }
}



abstract class: 不能直接 instantiated,只有部分已經實作的 methods


abstract class Animal {
    abstract makeSound(): void;
    move(): void {
        console.log("roaming the earth...");
    }
}

Iterators

for..of 及 for..in


for..in 會回傳 a list of keys
for..of 會回傳 a list of values


let list = [4, 5, 6];

for (let i in list) {
   console.log(i); // "0", "1", "2",
}

for (let i of list) {
   console.log(i); // "4", "5", "6"
}

Modules

自 ECMAScript 2015 開始,JavaScript 支援了 modules,modules 可讓 variables, functions, classes 運作在 modules 中,外面的程式只能 import 使用 export 的部分。


StringValidator.ts


export interface StringValidator {
    isAcceptable(s: string): boolean;
}

LettersOnlyValidator.ts


import { StringValidator } from "./StringValidator";

const lettersRegexp = /^[A-Za-z]+$/;

export class LettersOnlyValidator implements StringValidator {
    isAcceptable(s: string) {
        return lettersRegexp.test(s);
    }
}

ZipCodeValidator.ts


import { StringValidator } from "./StringValidator";

const numberRegexp = /^[0-9]+$/;

export class ZipCodeValidator implements StringValidator {
    isAcceptable(s: string) {
        return s.length === 5 && numberRegexp.test(s);
    }
}

// 定義 ZipCodeValidator 時沒有 export,可將 export 獨立寫成一行
//export { ZipCodeValidator };
// export 時,以 as 語法 rename class name
export { ZipCodeValidator as mainValidator };

AllValidators.ts


// 可將其他三個 ts 的 export 合併在一起

export * from "./StringValidator"; // exports interface 'StringValidator'
export * from "./LettersOnlyValidator"; // exports class 'LettersOnlyValidator'
export * from "./ZipCodeValidator";  // exports class 'ZipCodeValidator'

Test.ts


import { StringValidator } from "./StringValidator";
import { ZipCodeValidator } from "./ZipCodeValidator";
import { LettersOnlyValidator } from "./LettersOnlyValidator";

// Some samples to try
let strings = ["Hello", "98052", "101"];

// Validators to use
let validators: { [s: string]: StringValidator; } = {};
validators["ZIP code"] = new ZipCodeValidator();
validators["Letters only"] = new LettersOnlyValidator();

// Show whether each string passed each validator
strings.forEach(s => {
    for (let name in validators) {
        console.log(`"${ s }" - ${ validators[name].isAcceptable(s) ? "matches" : "does not match" } ${ name }`);
    }
});

namespaces

namespaces 為 internal modules,
modules 為 external modules。


將所有 validator 相關的程式放在 Validation 這個 namespace 裡面,但同樣要用 export 開放使用的介面。


namespace Validation {
    export interface StringValidator {
        isAcceptable(s: string): boolean;
    }

    const lettersRegexp = /^[A-Za-z]+$/;
    const numberRegexp = /^[0-9]+$/;

    export class LettersOnlyValidator implements StringValidator {
        isAcceptable(s: string) {
            return lettersRegexp.test(s);
        }
    }

    export class ZipCodeValidator implements StringValidator {
        isAcceptable(s: string) {
            return s.length === 5 && numberRegexp.test(s);
        }
    }
}

// Some samples to try
let strings = ["Hello", "98052", "101"];

// Validators to use
let validators: { [s: string]: Validation.StringValidator; } = {};
validators["ZIP code"] = new Validation.ZipCodeValidator();
validators["Letters only"] = new Validation.LettersOnlyValidator();

// Show whether each string passed each validator
for (let s of strings) {
    for (let name in validators) {
        console.log(`"${ s }" - ${ validators[name].isAcceptable(s) ? "matches" : "does not match" } ${ name }`);
    }
}

Declaration Files


在 TypeScript 要使用 JavaScript 的 libraries,必須要有該 libray 的定義檔


以 jQuery 為例,通常會使用 $('#id') 或是 jQuery('#id') 這樣的語法,在 TypeScript 編譯時,並不知道 $ 或是 jQuery 的意思,這時需要用 declare 語法定義 jQuery。


通常會把 Declaration File 放在獨立的檔案中,例如


// jQuery.d.ts

declare var jQuery: (string) => any;

然後再以 /// 語法引用


/// <reference path="./jQuery.d.ts" />

jQuery('#foo');

完整的 jQuery Declaration File 已經有人寫好了,可以直接下載 DefinitelyTyped/types/jquery/index.d.ts


但在 TypeScript 2.0+ 已經不建議這樣做,而是改用 @types 來管理所有 library 的 Declaration Files,可以用 npm 安裝 jquery 的 @types


npm install --save-dev @types/jquery

使用 jQuery


/// <reference types="jquery" />

$('#id');

編譯


tsc --target ES6 test.ts

如果要搜尋其他 libraries 的定義檔,可在這個網站 DefinitelyTyped: The repository for high quality TypeScript type definitions 搜尋


ref:


How to use jQuery with TypeScript


TypeScript 聲明文件


References


TypeScript新手入門


學習TypeScript:初體驗


我用 TypeScript 語言的七個月


How to use jQuery with TypeScript


Importing jqueryui with Typescript and RequireJS


Adding jQuery and jQueryUI to your TypeScript project




JS ECMAScript 6 compatibility table