Casto po vyreseni nejakeho mensiho problemu davam jeho reseni na GitHub GIST.
Na klasicky clanek to byva kratke, ale i tak je skoda se nepodelit. Proto budu postupne, jak cas dovoli prochazet stare gisty, trochu vic je komentovat a tvorit z nich kratke clanky.
Dnes zacnu posilanim HTTP POST pozadavku pomoci busybox telnetu.
K cemu je to dobre? Casto je potreba z pc odeslat data ke zpracovani na server. Pokud jde o bezny „stolni pc“, muzeme vyuzit nejaky pohodlny Python/Java framework, obetovat par desitek MB RAM a vse bude krasne fungovat 🙂
Ja ale potreboval posilat data z OpenWrt routeru, ktery mel 4MB flash a 16MB RAM. Proto jsem hledal reseni s nejmensimi naroky na zdroje.
Vetsinou se stejne pouzivaji HTTP pozadavky. Protoze jde o plain-text protokol, je mozne jednoduse pouzit telnet klienta. Bezni telnet klienti v Debianu/Ubuntu maji trochu jinou syntax, nez busybox klient.
Pro vetsi univerzalnost pouziti jsem pouzil prave busybox telnet.
Zde je ukazka na poslani JSONu pres HTTP GET. Na zacatku se vola program, jehoz vystupem je validni JSON.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 |
#!/bin/sh host=10.123.1.1 port=80 cmd_output=`/usr/bin/program_jehoz_vystupem_je_json -a -b -c parametry` ( echo open sleep 1 echo "GET /test HTTP/1.0" echo "Host: www.example.com" echo "Content-Type: application/json" echo "Cache-Control: no-cache" echo "Pragma: no-cache" echo "Connection: close" echo "" echo ${cmd_output} sleep 1 echo close ) | telnet $host $port |
Protoze program neumi zpracovat HTTP odpoved ani pokracovat v posilani pozadavku, posila se hlavicka „Connection: close„.
Data se poislaji vzdy aktualni, proto hlavicky na zakazani cache.
Po posledni hlavicce musi podle RFC popisujici HTTP nasledovat prazdny radek a pote samotna data.
Dost pravdepodobne bude mit server problem kvuli chybejici hlavicce Content-Length.
Ukazka nefunguje s busybox telnetem, pouze s plnohodnotnym telnetem v Debianu/Ubuntu.
Ukazka nefunguje s busybox telnetem, pouze s plnohodnotnym telnetem v Debianu/Ubuntu.
Vyse uvedeny kod ma nekolik problemu:
- Pohodlnejsi je posilani vytupu do telnetu pres rouru.
- Chybejici hlavicka Content-Length.
- Pokud je vystup prikazu hodne velky, nevejde se do promenne a cast se zahodi.
- Nepouziva kompresi, zbytecne plytvani pri prenosu.
- Nejde pres nej posilat binarni data.
Ad 1 & 2
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 |
#!/bin/sh # funguje s busybox telnetem i s normalnim telnetem v ubuntu # Pouziti: ./skript.sh | telnet 10.123.1.11 9876 #(ip port) cmd_output=`/usr/bin/program_jehoz_vystupem_je_json -a -b -c parametry` LENGTH=`echo $cmd_output | wc -c` # JE POTREBA MIT DOBRE Content-Length ! echo "POST /test HTTP/1.0" echo "Content-Type: application/json" echo "Cache-Control: no-cache" echo "Pragma: no-cache" echo "Connection: close" echo "Content-Length: $LENGTH" echo "" echo ${cmd_output} |
Bez spatne, nebo zadne hlavicky Content-Length bude mit hodne serveru pravdepodobne problem se zpracovanim! Pri mem testovani tomu tak bylo.
Ad 3
resenim je pouzit docasny soubor. Jeho nazev bychom si meli nechat pridelit od systemu, aby nedoslo ke kolizi:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 |
#!/bin/sh # Pouziti: ./skript.sh | telnet 10.123.1.11 9876 #(ip port) TMP=`mktemp -t` echo `/usr/bin/program_jehoz_vystupem_je_json -a -b -c parametry` >> $TMP LENGTH=`cat $TMP | wc -c` # JE POTREBA MIT DOBRE Content-Length ! echo "POST /test HTTP/1.0" echo "Content-Type: application/json" echo "Cache-Control: no-cache" echo "Pragma: no-cache" echo "Connection: close" echo "Content-Length: $LENGTH" echo "" echo `cat $TMP` echo rm $TMP |
Na zaver po sobe uklidime temp soubor.
Ad 4 & 5
body 4 a 5 spolu dost souvisi. Na komperesi pouzijeme standardni gzip, protoze je snad vzdy v busyboxu. Serveru proste http hlavickou oznamime jenom podporu gzipu.
Protoze gzip data jsou binarni, bude problem s odeslanim. Temer ve 100% pripadu telnet klient v binarnich datech rozpozna nejake ridici prikazy a spadne.
Resenim je zagzipovat data a pak je prevest do base64. I tak je uspora oproti plain-textu znacna. Pro srovnani pri mem testu: cisty text zabiral 10580 bajtu a base64-gzip 3056 bajtu – 3,46x mene.
Nyni prichazi dalsi problem 🙂
V beznych distribucich mame utilitu „base64“. Busybox ji take obsahuje, ale dost casto je kompilovan bez ni. Takze si musime base64 generovat samy, idealne v shellu – opet cisty POSIX sh shell.
Teoreticky to jde, ale nejde vubec o snadny ukol. Nastesti jsem nasel uz hotovou implementaci, jako bonus je pod public domain licenci.
Ze skriptu nas zajima pouze funkce encode(). Pro ukazku cast te magie:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 |
#... hexdump -v -e '2/1 "%02x"' | \ sed -e 's/0/0000 /g;s/1/0001 /g;s/2/0010 /g;s/3/0011 /g; s/4/0100 /g;s/5/0101 /g;s/6/0110 /g;s/7/0111 /g; s/8/1000 /g;s/9/1001 /g;s/a/1010 /g;s/b/1011 /g; s/c/1100 /g;s/d/1101 /g;s/e/1110 /g;s/f/1111 /g;' | \ tr -d ' ' | \ sed -e 's/[01]\{6\}/\0 /g' | \ sed -e 's_000000_A_g; s_000001_B_g; s_000010_C_g; s_000011_D_g; s_000100_E_g; s_000101_F_g; s_000110_G_g; s_000111_H_g; s_001000_I_g; s_001001_J_g; s_001010_K_g; s_001011_L_g; s_001100_M_g; s_001101_N_g; s_001110_O_g; s_001111_P_g; s_010000_Q_g; s_010001_R_g; s_010010_S_g; s_010011_T_g; #... |
Kompletni funkcni kod:
Vezme data, zkomprimuje gzipem a prevede do base64. Na zacatku souboru je mozne pomoci promenne GZIP urcit, zda se posle plain-text, nebo gzip+base64.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 |
#!/bin/sh # Pouziti: ./skript.sh | telnet 10.123.1.11 9876 #(ip port) GZIP=1 #GZIP=0 # encode() - autor "mateusza" - https://github.com/mateusza/shellscripthttpd/blob/master/base64.sh : encode(){ hexdump -v -e '2/1 "%02x"' | \ sed -e 's/0/0000 /g;s/1/0001 /g;s/2/0010 /g;s/3/0011 /g; s/4/0100 /g;s/5/0101 /g;s/6/0110 /g;s/7/0111 /g; s/8/1000 /g;s/9/1001 /g;s/a/1010 /g;s/b/1011 /g; s/c/1100 /g;s/d/1101 /g;s/e/1110 /g;s/f/1111 /g;' | \ tr -d ' ' | \ sed -e 's/[01]\{6\}/\0 /g' | \ sed -e 's_000000_A_g; s_000001_B_g; s_000010_C_g; s_000011_D_g; s_000100_E_g; s_000101_F_g; s_000110_G_g; s_000111_H_g; s_001000_I_g; s_001001_J_g; s_001010_K_g; s_001011_L_g; s_001100_M_g; s_001101_N_g; s_001110_O_g; s_001111_P_g; s_010000_Q_g; s_010001_R_g; s_010010_S_g; s_010011_T_g; s_010100_U_g; s_010101_V_g; s_010110_W_g; s_010111_X_g; s_011000_Y_g; s_011001_Z_g; s_011010_a_g; s_011011_b_g; s_011100_c_g; s_011101_d_g; s_011110_e_g; s_011111_f_g; s_100000_g_g; s_100001_h_g; s_100010_i_g; s_100011_j_g; s_100100_k_g; s_100101_l_g; s_100110_m_g; s_100111_n_g; s_101000_o_g; s_101001_p_g; s_101010_q_g; s_101011_r_g; s_101100_s_g; s_101101_t_g; s_101110_u_g; s_101111_v_g; s_110000_w_g; s_110001_x_g; s_110010_y_g; s_110011_z_g; s_110100_0_g; s_110101_1_g; s_110110_2_g; s_110111_3_g; s_111000_4_g; s_111001_5_g; s_111010_6_g; s_111011_7_g; s_111100_8_g; s_111101_9_g; s_111110_+_g; s_111111_/_g; s_0000_A=_g; s_0001_E=_g; s_0010_I=_g; s_0011_M=_g; s_0100_Q=_g; s_0101_U=_g; s_0110_Y=_g; s_0111_c=_g; s_1000_g=_g; s_1001_k=_g; s_1010_o=_g; s_1011_s=_g; s_1100_w=_g; s_1101_0=_g; s_1110_4=_g; s_1111_8=_g; s_00_A==_; s_01_Q==_; s_10_g==_; s_11_w==_; ' | \ tr -d ' ' | \ sed -e 's/.\{64\}/\0\n/g' echo } TMP=`mktemp -t` echo `/usr/bin/program_jehoz_vystupem_je_json -a -b -c parametry` >> $TMP if [ $GZIP -eq 1 ]; then gzip $TMP cat $TMP.gz | encode > $TMP.gz.base64 LENGTH=`cat $TMP.gz.base64 | wc -c` # JE POTREBA MIT DOBRE Content-Length ! else LENGTH=`cat $TMP | wc -c` # JE POTREBA MIT DOBRE Content-Length ! fi echo "POST /test/abc HTTP/1.0" echo "User-Agent: wtf/1.0" echo "Content-Type: application/json" if [ $GZIP -eq 1 ]; then echo "Content-Encoding: gzip" echo "Content-Transfer-Encoding: base64" fi echo "Cache-Control: no-cache" echo "Pragma: no-cache" echo "Connection: close" echo "Content-Length: $LENGTH" echo if [ $GZIP -eq 1 ]; then echo `cat $TMP.gz.base64` else echo `cat $TMP` fi echo rm $TMP* |
Aktualni verze bude vzdy na https://gist.github.com/tuxmartin/4d8d70dd8e6552d4db72