Files
smollora/__pycache__/prepare_data.cpython-312.pyc

73 lines
7.7 KiB
Plaintext
Raw Normal View History

<EFBFBD>
<00><><EFBFBD>i<EFBFBD><00><00><00>dZddlZddlZddlmZddlmZdZdZe d<07>dze d <09>zZ
e d<07>d
ze d <09>zZ e d<07>d ze d <09>zZ e d<07>d ze d <09>zZ d ZdZdZdZdd<11>Zd<12>Zd<13>Zd<14>Zedk(re<15>yy)a9
Prepare tool-calling training data for SmolLM3-3B LoRA fine-tuning.
Datasets:
1. interstellarninja/tool-calls-multiturn
2. NousResearch/Hermes-Function-Calling-V1
Both use ShareGPT format (from/value) with inline tool call tags.
We convert to SmolLM3's native token format.
Output: train.jsonl, val.jsonl
<EFBFBD>N)<01>Path)<01> load_datasetg<74><67><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD>?<3F>*<00><<00> tool_call<6C>>z
/tool_call<6C> tool_responsez/tool_responsez<|tool_call_start|>z<|tool_call_end|>z<|tool_response_start|>z<|tool_response_end|>c<00><><00>g}|rZ t|t<00>rtj|<01>n|}|r-dj d<03>|D<00><00>}d|<04>dt <00>dt<00>d<02>}nd}nd}|jd|d <09><02>|D<00>]}|jd
|jd d <0C><00>}|jd |jdd <0C><00>}|dk(r<01>N|dvr|jd|d <09><02><00>g|dvrZ|jtt <00>} | jtt<00>} | j<00>s<01><>|jd| d <09><02><00><>|dk(s<01><>|jtt<00>} | jt t"<00>} |jd| d <09><02><00><01>t%d<14>|D<00><00>}
|
sy|S#tj$rd}Y<00><01><>wxYw)z8Convert ShareGPT-style conversation to SmolLM3 messages.N<>
c3<00>JK<00>|]}tj|d<00><01><00><01><00>y<02>w)F<><01> ensure_asciiN)<02>json<6F>dumps)<02>.0<EFBFBD>ts <20>+/home/openclaw/dev/smollora/prepare_data.py<70> <genexpr>z.convert_sharegpt_to_smollm3.<locals>.<genexpr>0s<00><00><><00>!X<><01>$<24>*<2A>*<2A>Q<EFBFBD>U<EFBFBD>"C<>"C<>!X<>s<00>!#z<>You are a helpful AI assistant named SmolLM, trained by Hugging Face.
### Tools
You may call one or more functions to assist with the user query.
You are provided with function signatures within <tools></tools> XML tags:
<tools>
zn
</tools>
For each function call, return a json object with function name and arguments within special tags:
z>
{{"name": <function-name>, "arguments": <args-json-object>}}
zEYou are a helpful AI assistant named SmolLM, trained by Hugging Face.<2E>system)<02>role<6C>content<6E>fromr<00><00>valuer)<02>human<61>userr)<02> assistant<6E>gptr<00>toolc3<00>\K<00>|]$}|ddk(rt|jdd<03>v<00><01><00>&y<04>w)rrrrN)<02> SMOL_TC_START<52>get)r<00>ms rrz.convert_sharegpt_to_smollm3.<locals>.<genexpr>Us4<00><00><><00><06> <0A> <0C>V<EFBFBD>9<EFBFBD> <0B> #<23> <16><11><15><15>y<EFBFBD>"<22>-<2D>-<2D><06>s<00>*,)<13>
isinstance<EFBFBD>strr<00>loads<64>JSONDecodeError<6F>joinr!<00> SMOL_TC_END<4E>appendr"<00>replace<63>TC_OPEN<45>TC_CLOSE<53>strip<69>TR_OPEN<45> SMOL_TR_START<52>TR_CLOSE<53> SMOL_TR_END<4E>any) <0B> conversations<6E>
tools_json<EFBFBD>messages<65>
tools_list<EFBFBD> tool_defs<66>system_content<6E>turnrrr<00> has_tool_calls r<00>convert_sharegpt_to_smollm3r<%s<><00><00><11>H<EFBFBD><11> <1E>3=<3D>j<EFBFBD>#<23>3N<33><14><1A><1A>J<EFBFBD>/<2F>T^<5E>J<EFBFBD> <16><1C> <09> <09>!X<>Z<EFBFBD>!X<>X<>I<EFBFBD><1C>&<26>;<3B>'"<22>"/<2F><1F>1Q<01><1E>-<2D>r<EFBFBD>#<23> <1B>e<01>N<EFBFBD>`<60><0E> <0C>O<EFBFBD>O<EFBFBD>X<EFBFBD>.<2E>A<>B<><1D>B<01><04><13>x<EFBFBD>x<EFBFBD><06><04><08><08><16><12> 4<>5<><04><14><08><08><17>$<24>(<28>(<28>9<EFBFBD>b<EFBFBD>"9<>:<3A><05> <0F>8<EFBFBD> <1B> <14> <11>&<26> &<26> <14>O<EFBFBD>O<EFBFBD>V<EFBFBD><05>><3E> ?<3F> <11>)<29> )<29><1B>m<EFBFBD>m<EFBFBD>G<EFBFBD>]<5D>;<3B>G<EFBFBD><1D>o<EFBFBD>o<EFBFBD>h<EFBFBD> <0B><<3C>G<EFBFBD><16>}<7D>}<7D><EFBFBD><18><0F><0F><1B><17> I<>J<> <11>V<EFBFBD>^<5E><1B>m<EFBFBD>m<EFBFBD>G<EFBFBD>]<5D>;<3B>G<EFBFBD><1D>o<EFBFBD>o<EFBFBD>h<EFBFBD> <0B><<3C>G<EFBFBD> <14>O<EFBFBD>O<EFBFBD>V<EFBFBD><07>@<40> A<>!B<01>$<18><06><19><06><06>M<EFBFBD>
<19><13> <13>O<EFBFBD><4F>c<14>#<23>#<23> <1E><1D>J<EFBFBD> <1E>s<00>'F$<00>$F=<03><F=c<00>^<00>td<01>tdd<03><04>}g}|D]u}|jdg<00>}|jd<06>}|s<01>)t|t<00>r|n|rt j |<04>nd}t||<05>}|s<01>c|jd|i<01><00>wtdt|<01><00>d <09><03>|S)
Nz2Loading interstellarninja/tool-calls-multiturn ...z&interstellarninja/tool-calls-multiturn<72>train<69><01>splitr4<00>toolsr6<00> -> <20> samples with tool calls<6C>
<EFBFBD>printrr"r$r%rrr<r*<00>len<65><07>ds<64>samples<65>rowr4rA<00> tools_str<74> converteds r<00>load_multiturn_datasetrM`s<><00><00> <09>
><3E>?<3F> <15>><3E>g<EFBFBD> N<>B<EFBFBD><10>G<EFBFBD><11>4<><03><1B><07><07><0F><12>4<> <0A><13><07><07><07> <20><05><1C> <14>'<27><05>s<EFBFBD>3<>E<EFBFBD>u<EFBFBD>$<24>*<2A>*<2A>U<EFBFBD>:K<>Z^<5E> <09>/<2F> <0A>y<EFBFBD>I<> <09> <14> <13>N<EFBFBD>N<EFBFBD>J<EFBFBD> <09>2<> 3<>4<>
<EFBFBD>E<EFBFBD>#<23>g<EFBFBD>,<2C><1E>7<>
8<EFBFBD>9<> <12>N<EFBFBD>c<00>^<00>td<01>tdd<03><04>}g}|D]u}|jdg<00>}|jd<06>}|s<01>)t|t<00>r|n|rt j |<04>nd}t||<05>}|s<01>c|jd|i<01><00>wtdt|<01><00>d <09><03>|S)
Nz3Loading NousResearch/Hermes-Function-Calling-V1 ...z'NousResearch/Hermes-Function-Calling-V1r>r?r4rAr6rBrCrDrGs r<00>load_hermes_fc_datasetrPqs<><00><00> <09>
?<3F>@<40> <15>?<3F>w<EFBFBD> O<>B<EFBFBD><10>G<EFBFBD><11>4<><03><1B><07><07><0F><12>4<> <0A><13><07><07><07> <20><05><1C> <14>'<27><05>s<EFBFBD>3<>E<EFBFBD>u<EFBFBD>$<24>*<2A>*<2A>U<EFBFBD>:K<>Z^<5E> <09>/<2F> <0A>y<EFBFBD>I<> <09> <14> <13>N<EFBFBD>N<EFBFBD>J<EFBFBD> <09>2<> 3<>4<>
<EFBFBD>E<EFBFBD>#<23>g<EFBFBD>,<2C><1E>7<>
8<EFBFBD>9<> <12>NrNc <00><><00>ddl}|j<00>}|jdtd<03><04>|jdtdd<06><07>|j <00>}t |j<00>}|jdd<08> <09>g}|jt<00><00>|jt<00><00>td
t|<04><00><00><02>tjt <00>tj"|<04>|j$dkDr|d|j$}t'd t t|<04>t(z<00><00>}|d|}||d}td t|<07><00>d t|<06><00><00><04>d|fd|ffD]a\}} ||<08>d<10>z }
t+|
d<11>5} | D]+} | j-t/j0| d<12><13>dz<00><00>- ddd<00>td|
<EFBFBD><00><02><00>ctd<16>y#1swY<00>%xYw)Nrz --output-dirz/data)<02>type<70>defaultz --max-sampleszLimit total samples (0=all))rRrS<00>helpT)<02>parents<74>exist_okz
Total raw samples: <20>zTrain: z, Val: r><00>valz.jsonl<6E>wFr r zWrote zData preparation complete!)<19>argparse<73>ArgumentParser<65> add_argumentr%<00>int<6E>
parse_argsr<00>
output_dir<EFBFBD>mkdir<69>extendrMrPrErF<00>random<6F>seed<65>SEED<45>shuffle<6C> max_samples<65>max<61> VAL_FRACTION<4F>open<65>writerr) rZ<00>parser<65>argsr_<00> all_samples<65> val_count<6E> val_samples<65> train_samples<65>
split_name<EFBFBD>
split_data<EFBFBD>path<74>f<>ss r<00>mainrv<00>s<><00><00><13> <15> $<24> $<24> &<26>F<EFBFBD>
<EFBFBD><17><17><0E>S<EFBFBD>'<27><17>B<>
<EFBFBD><17><17><0F>c<EFBFBD>1<EFBFBD>C`<60><17>a<> <11> <1C> <1C> <1E>D<EFBFBD><15>d<EFBFBD>o<EFBFBD>o<EFBFBD>&<26>J<EFBFBD><0E><14><14>T<EFBFBD>D<EFBFBD><14>1<><14>K<EFBFBD><0F><16><16>-<2D>/<2F>0<><0F><16><16>-<2D>/<2F>0<> <09> !<21>#<23>k<EFBFBD>"2<>!3<>
4<EFBFBD>5<>
<EFBFBD>K<EFBFBD>K<EFBFBD><04><15>
<EFBFBD>N<EFBFBD>N<EFBFBD>;<3B><1F> <0B><17><17>!<21><1B>!<21>"3<>4<EFBFBD>#3<>#3<>4<> <0B><13>A<EFBFBD>s<EFBFBD>3<EFBFBD>{<7B>+<2B>l<EFBFBD>:<3A>;<3B><<3C>I<EFBFBD><1D>j<EFBFBD>y<EFBFBD>)<29>K<EFBFBD><1F> <09>
<EFBFBD>+<2B>M<EFBFBD> <09>G<EFBFBD>C<EFBFBD> <0A>&<26>'<27>w<EFBFBD>s<EFBFBD>;<3B>/?<3F>.@<40>
A<EFBFBD>B<>$+<2B>]<5D>#;<3B>e<EFBFBD>[<5B>=Q<>"R<><1F><1E>
<EFBFBD>J<EFBFBD><19>z<EFBFBD>l<EFBFBD>&<26>1<>1<><04> <11>$<24><03>_<EFBFBD> B<01><01><1F> B<01><01><11><07><07><04>
<EFBFBD>
<EFBFBD>1<EFBFBD>5<EFBFBD>9<>D<EFBFBD>@<40>A<> B<01> B<01> <0E><06>t<EFBFBD>f<EFBFBD>o<EFBFBD><1E> <1F>
<EFBFBD>
&<26>'<27>  B<01> B<01>s <00> 1G<05>G( <09>__main__)N)<17>__doc__rrb<00>pathlibr<00>datasetsrrhrd<00>chrr,r-r/r1r!r)r0r2r<rMrPrv<00>__name__<5F>rNr<00><module>r~s<><00><01> <04> <0C> <0A><18>!<21><13> <0C> <09><04> <0E>b<EFBFBD>'<27>K<EFBFBD>
<1F>#<23>b<EFBFBD>'<27>
)<29><07> <0E>r<EFBFBD>7<EFBFBD>\<5C> !<21>C<EFBFBD><02>G<EFBFBD> +<2B><08>
<0A>b<EFBFBD>'<27>O<EFBFBD>
#<23>c<EFBFBD>"<22>g<EFBFBD>
-<2D><07> <0E>r<EFBFBD>7<EFBFBD>%<25> %<25><03>B<EFBFBD><07> /<2F><08>&<26> <0A>!<21> <0B>)<29> <0A>%<25> <0B>8<14>v<13>"<13>"#(<28>L <0C>z<EFBFBD><19><08>F<EFBFBD>rN