目 录CONTENT

文章目录

【实践】windows系统flink集群搭建

FatFish1
2025-05-05 / 0 评论 / 0 点赞 / 24 阅读 / 0 字 / 正在检测是否收录...

使用windows快捷搭建起一个单机flink

首先找到apache.flink的镜像包官网,选择需要的版本

https://archive.apache.org/dist/flink/

解压到本地后获得一个flink目录,找到bin目录,可以看到flink的启动文件

但是缺少在windows系统下可以使用的.bat文件,在网上找相关资料,找到如下内容,只需要创建并导入这些文件内容即可:

flink.bat文件如下:

::###############################################################################
::  Licensed to the Apache Software Foundation (ASF) under one
::  or more contributor license agreements.  See the NOTICE file
::  distributed with this work for additional information
::  regarding copyright ownership.  The ASF licenses this file
::  to you under the Apache License, Version 2.0 (the
::  "License"); you may not use this file except in compliance
::  with the License.  You may obtain a copy of the License at
::
::      http://www.apache.org/licenses/LICENSE-2.0
::
::  Unless required by applicable law or agreed to in writing, software
::  distributed under the License is distributed on an "AS IS" BASIS,
::  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
::  See the License for the specific language governing permissions and
:: limitations under the License.
::###############################################################################
 
@echo off
setlocal
 
SET bin=%~dp0
SET FLINK_HOME=%bin%..
SET FLINK_LIB_DIR=%FLINK_HOME%\lib
SET FLINK_PLUGINS_DIR=%FLINK_HOME%\plugins
 
SET JVM_ARGS=-Xmx512m
 
SET FLINK_JM_CLASSPATH=%FLINK_LIB_DIR%\*
 
java %JVM_ARGS% -cp "%FLINK_JM_CLASSPATH%"; org.apache.flink.client.cli.CliFrontend %*
 
endlocal

start-cluster.bat文件如下:

::###############################################################################
::  Licensed to the Apache Software Foundation (ASF) under one
::  or more contributor license agreements.  See the NOTICE file
::  distributed with this work for additional information
::  regarding copyright ownership.  The ASF licenses this file
::  to you under the Apache License, Version 2.0 (the
::  "License"); you may not use this file except in compliance
::  with the License.  You may obtain a copy of the License at
::
::      http://www.apache.org/licenses/LICENSE-2.0
::
::  Unless required by applicable law or agreed to in writing, software
::  distributed under the License is distributed on an "AS IS" BASIS,
::  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
::  See the License for the specific language governing permissions and
:: limitations under the License.
::###############################################################################
 
@echo off
setlocal EnableDelayedExpansion
 
SET bin=%~dp0
SET FLINK_HOME=%bin%..
SET FLINK_LIB_DIR=%FLINK_HOME%\lib
SET FLINK_PLUGINS_DIR=%FLINK_HOME%\plugins
SET FLINK_CONF_DIR=%FLINK_HOME%\conf
SET FLINK_LOG_DIR=%FLINK_HOME%\log
 
SET JVM_ARGS=-Xms1024m -Xmx1024m
 
SET FLINK_CLASSPATH=%FLINK_LIB_DIR%\*
 
SET logname_jm=flink-%username%-jobmanager.log
SET logname_tm=flink-%username%-taskmanager.log
SET log_jm=%FLINK_LOG_DIR%\%logname_jm%
SET log_tm=%FLINK_LOG_DIR%\%logname_tm%
SET outname_jm=flink-%username%-jobmanager.out
SET outname_tm=flink-%username%-taskmanager.out
SET out_jm=%FLINK_LOG_DIR%\%outname_jm%
SET out_tm=%FLINK_LOG_DIR%\%outname_tm%
 
SET log_setting_jm=-Dlog.file="%log_jm%" -Dlogback.configurationFile=file:"%FLINK_CONF_DIR%/logback.xml" -Dlog4j.configuration=file:"%FLINK_CONF_DIR%/log4j.properties"
SET log_setting_tm=-Dlog.file="%log_tm%" -Dlogback.configurationFile=file:"%FLINK_CONF_DIR%/logback.xml" -Dlog4j.configuration=file:"%FLINK_CONF_DIR%/log4j.properties"
 
:: Log rotation (quick and dirty)
CD "%FLINK_LOG_DIR%"
for /l %%x in (5, -1, 1) do ( 
SET /A y = %%x+1 
RENAME "%logname_jm%.%%x" "%logname_jm%.!y!" 2> nul
RENAME "%logname_tm%.%%x" "%logname_tm%.!y!" 2> nul
RENAME "%outname_jm%.%%x" "%outname_jm%.!y!"  2> nul
RENAME "%outname_tm%.%%x" "%outname_tm%.!y!"  2> nul
)
RENAME "%logname_jm%" "%logname_jm%.0"  2> nul
RENAME "%logname_tm%" "%logname_tm%.0"  2> nul
RENAME "%outname_jm%" "%outname_jm%.0"  2> nul
RENAME "%outname_tm%" "%outname_tm%.0"  2> nul
DEL "%logname_jm%.6"  2> nul
DEL "%logname_tm%.6"  2> nul
DEL "%outname_jm%.6"  2> nul
DEL "%outname_tm%.6"  2> nul
 
for %%X in (java.exe) do (set FOUND=%%~$PATH:X)
if not defined FOUND (
    echo java.exe was not found in PATH variable
    goto :eof
)
 
echo Starting a local cluster with one JobManager process and one TaskManager process.
 
echo You can terminate the processes via CTRL-C in the spawned shell windows.
 
echo Web interface by default on http://localhost:8081/.
 
start /b java %JVM_ARGS% %log_setting_jm% -cp "%FLINK_CLASSPATH%"; org.apache.flink.runtime.entrypoint.StandaloneSessionClusterEntrypoint --configDir "%FLINK_CONF_DIR%" > "%out_jm%" 2>&1
start /b java %JVM_ARGS% %log_setting_tm% -cp "%FLINK_CLASSPATH%"; org.apache.flink.runtime.taskexecutor.TaskManagerRunner --configDir "%FLINK_CONF_DIR%" > "%out_tm%" 2>&1
 
endlocal
 

然后在cmd中即可启动flink集群,首先进入flink安装目录的bin路径,执行:

start-cluster
--------------------------------------------------------------------------------
Starting a local cluster with one JobManager process and one TaskManager process.
You can terminate the processes via CTRL-C in the spawned shell windows.
Web interface by default on http://localhost:8081/.

看到如下输出,则启动成功,执行jps命令,看到:

179008 StandaloneSessionClusterEntrypoint

同时在浏览器中输入localhost:8081即可登录flink管理后台

但是在windows下我没有找到stop-cluster.bat,只能先通过taskkill -f -pid 179008来完成集群停止了

使用windows+cygwin快捷搭建起一个单机flink

有时开发人员还是更喜欢使用linux系统,不想使用虚拟机环境,也不想开云服务器的情况下,可以使用cygwin搭建

找到cygwin的官方安装地址

https://www.cygwin.com/install.html

下载后打开可执行文件

选择一个合适的根目录,这就是以后cyg虚拟环境使用的root目录,包括linux环境和命令安装都可以从这里进行

补链接图https://blog.csdn.net/weixin_42837669/article/details/114381405

然后到了这一步是要选择需要安装的命令,这里命令非常多,可以只选一些常用的,比如vim、curl等,当然,这里也可以不装命令,先把环境搞起来,后续想要重新安装命令,只需要打开cygwin的安装文件,再次回到这一步,挑选一些命令重新执行安装即可,不会影响当前环境内容

补链接图https://blog.csdn.net/weixin_42837669/article/details/114381405

安装好后即可打开我们的linux环境,这时候我们可以先开启一些命令别名,比如ll命令,在~/.bashrc文件下,是一个隐藏文件,使用ls -a命令可以找到,找到:

# alias ll='ls -l'

把这一行的注释解开即可,重启下cygwin的GUI,再打开就可以使用ll命令了

搭建好flink集群后,可以把之前下好的flink目录整体copy到我们自己的路径下,然后还是进入bin目录,这次不需要使用.bat文件了,直接执行以下命令:

./start-cluster.sh

即可启动集群

如果启动没成功,可以去../log目录下面看看日志,可能有这样的报错:

Improperly specified VM option 'MaxMetaspaceSize=268435456
Error: Could not create the Java Virtual Machine.
Error: A fatal exception has occurred. Program will exit.

只需要在config.sh里面注释掉如下代码即可:

export JVM_ARGS="${JVM_ARGS} ${jvm_params}"

再次启动就可以了,同样可以使用localhost:8081即可登录flink管理后台

启动单机flink的examples

可以看到,在flink的expamples目录下面有很多样例jar,选一个有代表性的/examples/streaming/WordCount.jar

执行以下命令:

flink run WordCount.jar

---------------------------
-bash: flink: command not found

解决这个问题的办法有两种,要么去bin目录下面执行./flink run ../examples/streaming/WordCount.jar

要么可以加环境变量解决

环境变量的本质作用是,当执行一个命令,系统先去环境变量所对应的目录下面寻找是否有这个命令

那么我们加以下两个变量:

FLINK_HOME        xxxxx\flink-1.15.0\bin
Path   xxxxxxxxxxxxx;%FLINK_HOME%

再次执行,如果发现有以下报错:

module java.base does not “opens java.util“ to unnamed module

可能是java版本太高,暂未找到在linux环境下的解决方案,换个jdk8版本再试试

Executing example with default input data.
Use --input to specify file input.
Printing result to stdout. Use --output to specify output path.
Job has been submitted with JobID 7abd1971de905dcc139b55d61333f6db

------------------------------------------------------------
 The program finished with the following exception:
  ……
Caused by: java.util.concurrent.CompletionException: org.apache.flink.runtime.jobmanager.scheduler.NoResourceAvailableException: Could not acquire the minimum require
d resources.
        at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:292)
        at java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:308)
        at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:607)
        at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:591)
        ... 37 more
Caused by: org.apache.flink.runtime.jobmanager.scheduler.NoResourceAvailableException: Could not acquire the minimum required resources.

这里又报错了,看上去好像是资源不够,流跑不起来

0

评论区