smollora/__pycache__/prepare_data.cpython-312.pyc

<EFBFBD>
<00><><EFBFBD>i<EFBFBD><00><00>
<00>dZd
dl
Z
d
dlZd
dlmZ
d
dlmZ
dZdZe	d<07>
dze	d	<09>
zZ
e	d<07>
d
ze	d	<09>
zZe	d<07>
dze	d	<09>
zZe	d<07>
dze	d	<09>
zZ
d
ZdZdZdZdd<11>
Zd<12>Zd<13>Zd<14>Zedk(re<15>
yy)a9

Prepare tool-calling training data for SmolLM3-3B LoRA fine-tuning.

Datasets:
  1. interstellarninja/tool-calls-multiturn
  2. NousResearch/Hermes-Function-Calling-V1

Both use ShareGPT format (from/value) with inline tool call tags.
We convert to SmolLM3's native token format.

Output: train.jsonl, val.jsonl
<EFBFBD>N)
<01>Path)
<01>load_datasetg<74><67><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD>?<3F>*<00><<00>	tool_call<6C>>z
/tool_call<6C>
tool_responsez/tool_responsez<|tool_call_start|>z<|tool_call_end|>z<|tool_response_start|>z<|tool_response_end|>c<00><><00>g}|
rZ	t
|
t<00>rtj|
<01>
n
|
}|r-djd<03>|D<00><00>
}d|<04>dt<00>dt<00>d<02>}nd}nd}|jd|d	<09><02>

|D<00>
]}|jd
|jdd<0C><00>}|jd
|jdd<0C><00>}|dk(r
<01>N|dvr|jd|d	<09><02>

<00>g|dvrZ|jtt<00>}	|	jtt<00>}	|	j<00>s
<01><>|jd|	d	<09><02>

<00><>|dk(s
<01><>|jtt<00>}	|	jt t"<00>}	|jd|	d	<09><02>

<00>
<01>t%d<14>|D<00><00>
}
|
s
y
|S#tj$r
d
}Y<00>
<01><>wxYw
)z8Convert ShareGPT-style conversation to SmolLM3 messages.N<>

c
3<00>JK
<00>|]}
t
j|
d<00>
<01><00>
<01>

<00>y<02>w
)F<>
<01>ensure_asciiN)<02>json<6F>dumps)<02>.0<EFBFBD>
ts  <20>+/home/openclaw/dev/smollora/prepare_data.py<70>	<genexpr>z.convert_sharegpt_to_smollm3.<locals>.<genexpr>0s<00><00><><00>!X<>
<01>$<24>*<2A>*<2A>Q<EFBFBD>U<EFBFBD>"C<>"C<>!X<>s<00>!#
z<>You are a helpful AI assistant named SmolLM, trained by Hugging Face.

### Tools

You may call one or more functions to assist with the user query.
You are provided with function signatures within <tools></tools> XML tags:

<tools>
zn
</tools>

For each function call, return a json object with function name and arguments within special tags:
z>
{{"name": <function-name>, "arguments": <args-json-object>}}
zEYou are a helpful AI assistant named SmolLM, trained by Hugging Face.<2E>system)<02>role<6C>content<6E>fromr<00><00>valuer)<02>human<61>userr)<02>	assistant<6E>gptr<00>toolc
3<00>\K
<00>|]$}
|
dd
k(rt|
jdd<03>v<00>
<01>

<00>&y<04>w
)rrrrN)<02>
SMOL_TC_START<52>get)r<00>
ms  rrz.convert_sharegpt_to_smollm3.<locals>.<genexpr>Us4<00><00><><00><06>
<0A><0C>V<EFBFBD>9<EFBFBD><0B>#<23>	<16><11><15><15>y<EFBFBD>"<22>-<2D>-<2D><06>s<00>*,
)<13>
isinstance<EFBFBD>strr<00>loads<64>JSONDecodeError<6F>joinr!<00>SMOL_TC_END<4E>appendr"<00>replace<63>TC_OPEN<45>TC_CLOSE<53>strip<69>TR_OPEN<45>
SMOL_TR_START<52>TR_CLOSE<53>SMOL_TR_END<4E>any)<0B>
conversations<6E>
tools_json<EFBFBD>messages<65>
tools_list<EFBFBD>	tool_defs<66>system_content<6E>turnrrr<00>
has_tool_calls           r<00>convert_sharegpt_to_smollm3r<%s<>
<00><00><11>H<EFBFBD><11>	<1E>3=<3D>j<EFBFBD>#<23>3N<33><14><1A><1A>J<EFBFBD>/<2F>T^<5E>J<EFBFBD><16><1C>	<09>	<09>!X<>Z<EFBFBD>!X<>X<>I<EFBFBD><1C>&<26>;<3B>'"<22>"/<2F><1F>
1Q
<01><1E>-<2D>r<EFBFBD>#<23>
<1B>e
<01>N<EFBFBD>`<60><0E><0C>O<EFBFBD>O<EFBFBD>X<EFBFBD>.<2E>A<>B<><1D>B
<01><04><13>x<EFBFBD>x<EFBFBD><06><04><08><08><16><12> 4<>5<><04><14><08><08><17>$<24>(<28>(<28>9<EFBFBD>b<EFBFBD>"9<>:<3A><05><0F>8<EFBFBD><1B><14>
<11>&<26>
&<26><14>O<EFBFBD>O<EFBFBD>V<EFBFBD><05>><3E>?<3F>
<11>)<29>
)<29><1B>m<EFBFBD>m<EFBFBD>G<EFBFBD>]<5D>;<3B>G<EFBFBD><1D>o<EFBFBD>o<EFBFBD>h<EFBFBD><0B><<3C>G<EFBFBD><16>}<7D>}<7D><EFBFBD><18><0F><0F><1B><17> I<>J<>
<11>V<EFBFBD>^<5E><1B>m<EFBFBD>m<EFBFBD>G<EFBFBD>]<5D>;<3B>G<EFBFBD><1D>o<EFBFBD>o<EFBFBD>h<EFBFBD><0B><<3C>G<EFBFBD><14>O<EFBFBD>O<EFBFBD>V<EFBFBD><07>@<40>A<>!B
<01>$<18><06><19><06><06>M<EFBFBD>
<19><13><13>O<EFBFBD><4F>c
<14>#<23>#<23>
	<1E><1D>J<EFBFBD>
	<1E>s<00>'F$<00>$F=<03><
F=c<00>^
<00>t
d
<01>

tdd<03><04>}g}
|D]u}|jdg<00>}|jd<06>
}|s
<01>)t|t<00>r|n|rtj|<04>
n
d}t||<05>}|s
<01>c|
jd|i
<01>

<00>wt
dt|
<01>
<00>d	<09><03>

|
S)
Nz2Loading interstellarninja/tool-calls-multiturn ...z&interstellarninja/tool-calls-multiturn<72>train<69>
<01>splitr4<00>toolsr6<00>  -> <20> samples with tool calls<6C>
<EFBFBD>printrr"r$r%rrr<r*<00>len<65><07>ds<64>samples<65>rowr4rA<00>	tools_str<74>	converteds       r<00>load_multiturn_datasetrM`s<><00><00>	<09>
><3E>?<3F>	<15>><3E>g<EFBFBD>	N<>B<EFBFBD><10>G<EFBFBD><11>4<><03><1B><07><07><0F><12>4<>
<0A><13><07><07><07> <20><05><1C><14>'<27><05>s<EFBFBD>3<>E<EFBFBD>u<EFBFBD>$<24>*<2A>*<2A>U<EFBFBD>:K<>Z^<5E>	<09>/<2F>
<0A>y<EFBFBD>I<>	<09><14><13>N<EFBFBD>N<EFBFBD>J<EFBFBD>	<09>2<>3<>4<>
<EFBFBD>E<EFBFBD>#<23>g<EFBFBD>,<2C><1E>7<>
8<EFBFBD>9<><12>N<EFBFBD>c<00>^
<00>t
d
<01>

tdd<03><04>}g}
|D]u}|jdg<00>}|jd<06>
}|s
<01>)t|t<00>r|n|rtj|<04>
n
d}t||<05>}|s
<01>c|
jd|i
<01>

<00>wt
dt|
<01>
<00>d	<09><03>

|
S)
Nz3Loading NousResearch/Hermes-Function-Calling-V1 ...z'NousResearch/Hermes-Function-Calling-V1r>r?r4rAr6rBrCrDrGs       r<00>load_hermes_fc_datasetrPqs<><00><00>	<09>
?<3F>@<40>	<15>?<3F>w<EFBFBD>	O<>B<EFBFBD><10>G<EFBFBD><11>4<><03><1B><07><07><0F><12>4<>
<0A><13><07><07><07> <20><05><1C><14>'<27><05>s<EFBFBD>3<>E<EFBFBD>u<EFBFBD>$<24>*<2A>*<2A>U<EFBFBD>:K<>Z^<5E>	<09>/<2F>
<0A>y<EFBFBD>I<>	<09><14><13>N<EFBFBD>N<EFBFBD>J<EFBFBD>	<09>2<>3<>4<>
<EFBFBD>E<EFBFBD>#<23>g<EFBFBD>,<2C><1E>7<>
8<EFBFBD>9<><12>NrNc	<00><><00>d
dl}|j<00>}
|
jdtd<03><04>
|
jdtd
d<06><07>
|
j<00>}t
|j<00>
}|jdd<08>	<09>
g}|jt<00><00>

|jt<00><00>

td
t|<04>
<00><00><02>

tjt <00>

tj"|<04>

|j$d
kDr|d|j$}t'dt	t|<04>
t(z<00>
<00>}|d|}||d}tdt|<07>
<00>d
t|<06>
<00><00><04>

d|fd|ffD]a\}}	||<08>d<10>z}
t+|
d<11>5}|	D]+}|j-t/j0|d<12><13>dz<00>

<00>-	ddd<00>
td|
<EFBFBD><00><02>

<00>ctd<16>

y#1s
w
Y

<00>%xYw
)Nrz--output-dirz/data)<02>type<70>defaultz
--max-sampleszLimit total samples (0=all))rRrS<00>helpT)<02>parents<74>exist_okz
Total raw samples: <20>
zTrain: z, Val: r><00>valz.jsonl<6E>
wFr
rzWrote zData preparation complete!)<19>argparse<73>ArgumentParser<65>add_argumentr%<00>int<6E>
parse_argsr<00>
output_dir<EFBFBD>mkdir<69>extendrMrPrErF<00>random<6F>seed<65>SEED<45>shuffle<6C>max_samples<65>max<61>VAL_FRACTION<4F>open<65>writerr)
rZ<00>parser<65>argsr_<00>all_samples<65>	val_count<6E>val_samples<65>
train_samples<65>
split_name<EFBFBD>
split_data<EFBFBD>path<74>
f<>
ss
             r<00>mainrv<00>s<>
<00><00><13>
<15>
$<24>
$<24>
&<26>F<EFBFBD>
<EFBFBD><17><17><0E>S<EFBFBD>'<27><17>B<>
<EFBFBD><17><17><0F>c<EFBFBD>1<EFBFBD>C`<60><17>a<><11><1C><1C><1E>D<EFBFBD><15>d<EFBFBD>o<EFBFBD>o<EFBFBD>&<26>J<EFBFBD><0E><14><14>T<EFBFBD>D<EFBFBD><14>1<><14>K<EFBFBD><0F><16><16>-<2D>/<2F>0<><0F><16><16>-<2D>/<2F>0<>	<09>!<21>#<23>k<EFBFBD>"2<>!3<>
4<EFBFBD>5<>
<EFBFBD>K<EFBFBD>K<EFBFBD><04><15>
<EFBFBD>N<EFBFBD>N<EFBFBD>;<3B><1F><0B><17><17>!<21><1B>!<21>"3<>4<EFBFBD>#3<>#3<>4<><0B><13>A<EFBFBD>s<EFBFBD>3<EFBFBD>{<7B>+<2B>l<EFBFBD>:<3A>;<3B><<3C>I<EFBFBD><1D>j<EFBFBD>y<EFBFBD>)<29>K<EFBFBD><1F>	<09>
<EFBFBD>+<2B>M<EFBFBD>	<09>G<EFBFBD>C<EFBFBD>
<0A>&<26>'<27>w<EFBFBD>s<EFBFBD>;<3B>/?<3F>.@<40>
A<EFBFBD>B<>$+<2B>]<5D>#;<3B>e<EFBFBD>[<5B>=Q<>"R<><1F><1E>
<EFBFBD>J<EFBFBD><19>z<EFBFBD>l<EFBFBD>&<26>1<>1<><04>
<11>$<24><03>_<EFBFBD>	B
<01>
<01><1F>

B
<01>
<01><11><07><07><04>
<EFBFBD>
<EFBFBD>1<EFBFBD>5<EFBFBD>9<>D<EFBFBD>@<40>A<>

B
<01>	B
<01>	<0E><06>t<EFBFBD>f<EFBFBD>o<EFBFBD><1E><1F>
<EFBFBD>
&<26>'<27>	B
<01>	B
<01>s<00>	1G<05>G(	<09>__main__)
N)<17>__doc__rrb<00>pathlibr<00>datasetsrrhrd<00>chrr,r-r/r1r!r)r0r2r<rMrPrv<00>__name__<5F>rNr<00><module>r~
s<><00>


<01>
<04>
<0C>
<0A><18>!<21><13><0C>	<09><04><0E>b<EFBFBD>'<27>K<EFBFBD>
<1F>#<23>b<EFBFBD>'<27>
)<29><07><0E>r<EFBFBD>7<EFBFBD>\<5C>!<21>C<EFBFBD><02>G<EFBFBD>+<2B><08>

<0A>b<EFBFBD>'<27>O<EFBFBD>
#<23>c<EFBFBD>"<22>g<EFBFBD>
-<2D><07><0E>r<EFBFBD>7<EFBFBD>%<25>%<25><03>B<EFBFBD><07>/<2F><08>&<26>
<0A>!<21><0B>)<29>
<0A>%<25><0B>8
<14>v

<13>"
<13>"#
(<28>L
<0C>z<EFBFBD><19><08>F<EFBFBD>rN